Transformers and Attention in AI

Transformers are neural-network architectures designed to process relationships between elements in a sequence using a mechanism called attention. They are the foundation of most modern large language models and are also widely used for images, audio, video, and multimodal AI.

The key idea is simple: when processing one token, the model can determine which other tokens in the available context are most relevant and combine information from them. This ability lets transformers model dependencies across language efficiently while supporting highly parallel training at large scale.

Table of Contents

• Why Transformers Were Needed
• Attention: The Core Idea
  • Queries, Keys, and Values
  • Attention Scores
  • Weighted Context
• Self-Attention
• Multi-Head Attention
• Position and Token Order
• Inside a Transformer Layer
  • Feed-Forward Networks
  • Residual Connections and Normalization
• Transformer Architectures
  • Encoder-Only Models
  • Decoder-Only Models
  • Encoder-Decoder Models
• Why Transformers Scale
• Attention Cost and Long Contexts
• Transformers in Production
• Conclusion
  • More Articles to Read

Why Transformers Were Needed

Language is sequential, but important relationships are not always local. A word near the end of a paragraph may depend on information introduced hundreds of tokens earlier.

Earlier sequence models such as recurrent neural networks processed tokens primarily one step at a time. Information from previous steps was carried forward through an internal state.

Conceptually, processing followed a sequential dependency:

Token 1 → Token 2 → Token 3 → Token 4 → Token 5

This approach created two important problems. Long-range information could become difficult to preserve, and sequential computation limited how efficiently training could use highly parallel hardware.

LSTMs and related recurrent architectures improved long-range memory, but they still retained much of the sequential processing constraint.

Transformers introduced a different approach. Instead of carrying all relevant information through a sequential hidden state, attention lets tokens directly incorporate information from other tokens.

Consider:

The database server became overloaded after the traffic spike, so it started rejecting connections.

To build a useful representation of it, the model needs to associate that token with database server. Attention provides a mechanism for learning such relationships even when relevant tokens are separated by other text.

Attention: The Core Idea

Attention calculates how strongly one token should use information from other tokens. Instead of treating every part of the context equally, the model produces learned relevance scores.

Suppose the model processes:

The cache was full, so it started evicting keys.

When constructing the representation for it, information associated with cache may receive more attention than information associated with keys.

The mechanism is implemented using three learned representations commonly called query, key, and value.

Queries, Keys, and Values

Each token representation is transformed into three vectors:

• Query (Q) represents what information the current token is looking for.
• Key (K) represents what information a token can be matched by.
• Value (V) contains the information that can contribute to the resulting representation.

These vectors are produced using learned weight matrices. Conceptually:

query = token_embedding @ query_weights
key = token_embedding @ key_weights
value = token_embedding @ value_weights

There is no manually programmed rule saying that pronouns should search for nouns. Training learns query and key transformations that become useful for the model's objective.

This is an important distinction: attention relationships are learned from data rather than encoded as linguistic rules.

Attention Scores

The query for one token is compared with keys from other tokens. A common transformer implementation uses a dot product to measure compatibility.

For one query and key:

score = query @ key

A larger score indicates that the key is more relevant to the query according to the representations learned by the model.

In scaled dot-product attention, scores are divided by the square root of the key dimension before applying softmax:

import numpy as np


def softmax(values: np.ndarray) -> np.ndarray:
    shifted = values - np.max(values)
    exp = np.exp(shifted)
    return exp / exp.sum()


def attention_weights(
    query: np.ndarray,
    keys: np.ndarray,
) -> np.ndarray:
    dimension = keys.shape[-1]
    scores = query @ keys.T
    scores = scores / np.sqrt(dimension)

    return softmax(scores)

The scaling prevents dot products from becoming excessively large as vector dimensions increase. Softmax converts the scores into normalized weights.

Weighted Context

The attention weights determine how much information to take from each value vector.

Suppose a token produces these simplified attention weights:

The resulting representation is a weighted combination of the corresponding value vectors:

context = attention_weights @ values

If cache receives a high weight, information represented by that token contributes strongly to the result.

Actual transformer models perform these operations simultaneously across many tokens using optimized matrix operations rather than calculating each relationship independently in application code.

Self-Attention

Self-attention means that queries, keys, and values are derived from the same sequence. Every token can therefore build a new representation using information from other tokens in that sequence.

Consider:

The worker could not process the message because the database connection had timed out.

Different tokens may need different relationships. The representation for message may use information about worker and process, while timed out may depend strongly on database connection.

Self-attention produces contextual representations instead of treating each token as an isolated symbol.

This is why the same word can acquire different internal representations depending on context:

The application stores money in the bank.

The application server is located near the river bank.

The original token may begin from a similar embedding, but attention incorporates different surrounding information as the representation passes through transformer layers.

For autoregressive language generation, self-attention includes an additional restriction: a token must not see future tokens that have not been generated yet.

A causal mask prevents attention from accessing those future positions during training and inference. When predicting token 10, the model can use tokens 1 through 9 but not tokens 11 or 12.

Without this restriction, training could leak the answer by allowing the model to inspect future tokens it is supposed to predict.

Multi-Head Attention

A single attention calculation provides one way of relating tokens. Transformers commonly run several attention operations in parallel using different learned projections. These are called attention heads.

Each head has its own query, key, and value transformations. This allows different heads to learn different useful relationships.

For a code sequence, one head might become useful for relationships between a function call and its arguments, while another may capture relationships involving variables, types, or surrounding control flow. These interpretations are not explicitly assigned during training; useful patterns emerge from optimization.

The outputs from the heads are combined and projected back into the model's representation space.

A simplified implementation looks like:

def multi_head_attention(
    hidden_states,
    attention_heads,
    output_projection,
):
    head_outputs = [
        head(hidden_states)
        for head in attention_heads
    ]

    combined = concatenate(head_outputs)

    return combined @ output_projection

Multiple heads increase the model's ability to represent different relationships simultaneously, but they also add computation and parameters.

More attention heads do not automatically mean better quality. Head count, model width, depth, training data, and other architectural choices must work together.

Position and Token Order

Attention itself compares token representations without inherently understanding sequence order. Without additional positional information, the model would have difficulty distinguishing sequences containing the same tokens in different arrangements.

For example:

The service called the database.

and:

The database called the service.

contain many of the same tokens but express different relationships.

Transformers therefore incorporate positional information into token processing. Different architectures use techniques such as positional embeddings, sinusoidal encodings, or rotary positional representations.

The implementation differs, but the purpose is the same: provide information about where tokens occur and how positions relate to one another.

Position handling becomes especially important as context windows grow. A model trained primarily on shorter sequences may not automatically behave equally well when asked to process dramatically longer ones, even if the serving system technically accepts them.

Inside a Transformer Layer

Attention is the most recognizable transformer mechanism, but it is not the entire transformer. A transformer layer typically combines attention with additional neural-network operations.

At a high level, a token representation passes through attention, feed-forward computation, normalization, and residual paths. The resulting representation becomes input to the next layer.

Large models stack many such layers. Each layer refines the representations based on patterns learned during training.

Feed-Forward Networks

After attention mixes information between token positions, a feed-forward network performs additional transformations independently for each position.

A simplified version looks like:

def feed_forward(hidden, weights_1, bias_1, weights_2, bias_2):
    expanded = hidden @ weights_1 + bias_1
    activated = gelu(expanded)

    return activated @ weights_2 + bias_2

The intermediate representation is commonly expanded to a larger dimension before being projected back down. This gives the network additional capacity to transform information after attention has combined relevant context.

In large language models, feed-forward layers can contain a substantial portion of the model's parameters and computational cost. Focusing only on attention therefore gives an incomplete picture of transformer inference.

Residual Connections and Normalization

Deep neural networks can become difficult to optimize when information and gradients must pass through many transformations. Transformers use residual connections that allow information to bypass individual transformations and be added back into their outputs.

Conceptually:

hidden = hidden + attention(normalize(hidden))
hidden = hidden + feed_forward(normalize(hidden))

Normalization helps keep activations numerically well behaved, while residual connections improve information and gradient flow through deep networks.

Exact ordering varies among transformer architectures, but these mechanisms are fundamental to making very deep models practical to train.

Transformer Architectures

Transformers can be organized in several ways depending on the task. Three useful categories are encoder-only, decoder-only, and encoder-decoder architectures.

Encoder-Only Models

Encoder-style models build contextual representations of an input sequence. Because the complete input is already available, attention can generally use information from both earlier and later positions.

These representations are useful for tasks such as classification, semantic understanding, entity recognition, and feature extraction.

For example, a support-ticket system could encode a message and classify it into billing, account, or technical categories without generating a long textual response.

Decoder-Only Models

Decoder-only transformers are widely used for generative language models. They predict tokens autoregressively: each output token is generated based on tokens that came before it.

Causal masking ensures that position n cannot inspect later positions while predicting the next token.

Generation therefore behaves conceptually as:

Prompt → Token 1 → Token 2 → Token 3 → ...

This sequential dependency has major production consequences. Prompt processing can be highly parallelized, but output decoding still requires repeated forward computation as tokens are generated.

Most general-purpose conversational LLM applications encounter this architecture. Large Language Models (LLMs) covers token generation and application-level LLM behavior in more detail.

Encoder-Decoder Models

Encoder-decoder transformers separate input processing from output generation. The encoder creates representations of the source input, while the decoder generates output using both previously generated tokens and information from the encoder.

This architecture naturally fits sequence-to-sequence transformations such as translation, summarization, and structured transformation.

The decoder can use cross-attention, where its queries attend to keys and values produced by the encoder. Unlike self-attention, the information being attended to comes from a different representation sequence.

These categories describe architectural patterns rather than strict product boundaries. Modern AI systems often combine transformer components with additional modalities, retrieval systems, tools, and specialized model components.

Why Transformers Scale

One major advantage of transformers is that training computations can be expressed as large matrix operations that run efficiently on GPUs and other accelerators.

Recurrent networks introduce dependencies between sequence steps. Step 100 generally depends on the recurrent state produced at step 99, limiting parallel processing across the sequence.

During transformer training, attention for many token positions can instead be computed together because the complete training sequence is available. This makes much better use of highly parallel hardware.

Large matrix multiplications also map well to accelerator architectures. Increasing model width, layer count, training tokens, and compute therefore became practical at scales that were difficult for earlier sequence architectures.

Transformers also support distributed training, where model parameters, batches, or computation can be partitioned across many accelerators.

This scalability does not make training simple. Large models can require synchronization across devices, enormous memory capacity, high-bandwidth interconnects, checkpoint storage, and substantial power and infrastructure cost.

The transformer's success comes partly from model quality and partly from how effectively its computations can exploit modern hardware.

Attention Cost and Long Contexts

Standard self-attention has an important scaling limitation. Each token may compare its query with keys from every other relevant token in the sequence.

For a sequence of length n, the attention-score matrix contains roughly n × n relationships. This means standard attention computation and memory requirements can grow approximately quadratically with sequence length for this part of the model.

This simplified comparison describes standard full attention rather than the total cost of the entire model, but it explains why long-context processing is an important systems problem.

Modern implementations use techniques such as optimized attention kernels, grouped or multi-query attention, sparse attention patterns, sliding windows, and other architectural changes to reduce memory or computational pressure.

Application architecture matters as well. Sending 100,000 tokens to a model because the context window permits it is not necessarily efficient. If only 2,000 tokens are relevant, retrieval can reduce latency, cost, and irrelevant information.

The relationship between tokens, context limits, and application behavior is covered further in AI Tokens and Context Windows.

Transformers in Production

Production transformer inference is constrained by more than raw model computation. Model weights must fit into accelerator memory, prompts must be processed, generated tokens require repeated decoding, and concurrent requests compete for limited serving capacity.

For autoregressive LLMs, inference can be separated into two useful phases. Prefill processes the input tokens, while decoding generates output tokens sequentially.

A long prompt can make prefill expensive. A long response can make decoding expensive. The workload therefore matters when capacity planning: a service processing large documents behaves differently from one accepting short prompts and generating long answers.

During decoding, transformer servers commonly cache attention keys and values calculated for previous tokens. This KV cache avoids recomputing the complete token history for every new token.

The optimization is essential for efficient generation, but it consumes memory that grows with active sequence length and concurrent requests. A server may therefore run out of usable accelerator memory for additional requests even when raw compute is not fully saturated.

Batching can improve accelerator utilization by processing work from several requests together. However, aggressive batching can increase queueing latency, creating a trade-off between throughput and interactive responsiveness.

Model architecture also affects deployment choices. Quantization can reduce the memory required for model weights and may improve inference throughput, but aggressive precision reduction can affect quality. Splitting a model across several accelerators allows larger models to run but introduces communication overhead.

For externally hosted models, these infrastructure details may be hidden behind an API, but their effects remain visible through latency, rate limits, context pricing, output speed, and availability.

At the application layer, the strongest optimization is often avoiding unnecessary model work. Shorter relevant prompts, bounded output lengths, caching, retrieval, smaller models for simple tasks, and asynchronous processing can produce larger cost improvements than low-level inference tuning.

Transformer capacity should be planned around tokens and active sequences, not only HTTP requests per second. Two requests can consume dramatically different resources when one contains 500 input tokens and another contains 50,000.

Conclusion

Transformers process sequences by allowing tokens to selectively incorporate information from other tokens through attention. Queries and keys determine relevance, values provide information, multi-head attention captures different relationships, and positional representations preserve sequence structure.

Stacking attention with feed-forward networks, normalization, and residual connections creates deep models capable of learning highly complex representations. Their parallel training characteristics enabled modern AI to scale dramatically, but long contexts, model size, KV-cache memory, and sequential output generation create important production constraints.

Attention explains how information moves through a transformer; production engineering determines how much of that computation an application can afford. Understanding both sides is essential when designing systems around modern language models.

More Articles to Read

• Neural Networks and Deep Learning
• Large Language Models (LLMs)
• AI Tokens and Context Windows
• AI Embeddings