A transformer is a neural network architecture built around one idea: self-attention. Every position in a sequence can look at every other position in parallel, weighted by learned compatibility scores. That single change — from step-by-step recurrence to one-shot global attention — is what unlocked modern large language models, modern vision models, and the multi-modal systems sitting underneath most production GenAI today. The 2017 paper “Attention Is All You Need” introduced the design for machine translation; by 2026 it is the default substrate for almost every frontier model, regardless of modality.

If you have ever had a scoping conversation where “AI”, “deep learning”, “LLM”, and “GenAI” got used interchangeably for ninety minutes, transformers are the architectural fact that ties most of those terms together. We see the same pattern across our engagements: once a team understands what self-attention does and where its cost lives, the conversation about feasibility, latency, and infrastructure becomes much more grounded.

The transformer architecture: what self-attention actually does

The core of the architecture is the self-attention mechanism. Given a sequence of tokens — words, image patches, audio frames — each token computes a query, key, and value vector. The attention score between two tokens is the dot product of one’s query with the other’s key, normalised across the sequence. The output for each token is a weighted sum of value vectors, with weights given by those scores.

In practice this means a transformer can learn, during training, that “cat” and “sat” are tightly coupled in “The cat sat on the mat” even though “mat” sits closer in raw position. The same machinery, applied to image patches, lets a Vision Transformer (ViT) couple a foreground object with the background lighting cue that disambiguates it. The same machinery again, applied to audio frames, lets Whisper-class models couple a phoneme with the prosodic context several seconds away.

The full architecture wraps this attention layer with a few standard pieces: positional embeddings (since attention itself is permutation-invariant), residual connections, layer normalisation, and a feed-forward sub-layer per block. Original encoder-decoder transformers used both halves — encoder for the source sequence, decoder autoregressively generating the target. Most modern LLMs (GPT-class, Llama, Mistral) are decoder-only; most vision and embedding models (BERT, ViT, DINOv2) are encoder-only. The split matters less than the attention.

Why transformers replaced RNNs and CNNs as the default

To see why this architecture took over, it helps to look at what it replaced.

Architecture	Parallelism	Long-range dependencies	Inductive bias	Where it still wins
RNN / LSTM	Sequential per step	Weak — vanishing gradients on long contexts	Temporal recency	Tiny on-device sequence models, streaming with hard latency budgets
CNN	Parallel across spatial dims	Limited reach without deep stacks	Locality, translation equivariance on grids	Small-data vision, edge deployments where compute is fixed
Transformer	Parallel across sequence	Strong — global attention at every layer	Minimal; learned from data	Almost everything at scale: language, vision at scale, audio, multi-modal

RNNs and LSTMs were the default for sequence modelling before 2017 because they encode an obvious prior: process one token at a time, carry a hidden state forward. That prior is also their cost. The hidden state has to compress arbitrarily long history into a fixed-size vector, and gradients have to flow back through every step during training. Long contexts decay; the vanishing gradient problem is well-known, and the various LSTM and GRU gating tricks only partially solve it.

CNNs solve a different problem — local pattern detection on grids — and they remain excellent at it. But “what relates to what” in an image is not always local. A ViT can directly attend across the whole image at every layer; a CNN has to stack layers to expand its receptive field. On large datasets the transformer’s flexibility wins; on small datasets the CNN’s inductive bias still tends to generalise better.

The other reason transformers won is operational. Self-attention is embarrassingly parallel across the sequence dimension, which means it maps cleanly onto GPU and TPU hardware. RNNs leave most of a modern accelerator idle.

Where transformers sit in the broader ML taxonomy

This is the question that comes up most often in scoping conversations: where exactly does the transformer fit in the working taxonomy of symbolic AI, classical ML, deep learning, LLMs, and generative AI? The short answer is that transformer is an architecture — a kind of deep neural network — and it shows up across multiple categories of system:

• Discriminative deep learning. Encoder-only transformers (BERT, RoBERTa, ViT, DINOv2) used for classification, retrieval, embeddings. The model predicts labels or representations, not new content.
• Generative deep learning / LLMs. Decoder-only transformers (GPT-class, Llama, Mistral, Claude, Gemini) used to generate text autoregressively. These are what most people mean by “LLM” or “generative AI” today.
• Multi-modal systems. Transformer-based models that bind text to images (CLIP, LLaVA), text to audio (Whisper, AudioLM), or all three (Gemini-class, GPT-4o-class). The shared attention substrate is what makes the binding tractable.
• Reinforcement learning policies. Decision transformers and similar designs treat trajectory modelling as a sequence problem.

So “transformer” is not synonymous with “LLM” or “generative AI”, even though most current frontier LLMs and most current generative AI systems are transformer-based. It is the substrate; the loss function and training data determine which family of system you end up with.

What “long-range dependency” means in practice

The phrase “captures long-range dependencies” is used so often it loses meaning. Concretely:

1. In language, it means the model can resolve a pronoun against an antecedent fifty sentences earlier, or follow a chain of reasoning across a long document, without information being washed out by intermediate tokens.
2. In vision, it means the model can couple two regions of an image that are spatially distant — a hand in one corner, an object in another — without stacking dozens of convolutional layers.
3. In audio, it means the model can use prosodic or semantic cues from several seconds away to disambiguate a current phoneme or word.

Self-attention is what makes this cheap to learn, and it is also what makes inference expensive: the attention cost is quadratic in sequence length. A 4K-token context is fine; a 128K-token context is not, without optimisations.

The 2026 inference stack: why transformers are also cheap to serve

A point that often gets missed: transformers dominate partly because the inference stack around them has been hardware-optimised in ways the alternatives have not. Five years of engineering went into making decoder-only transformer inference fast on commodity GPUs.

• KV-cache. During autoregressive generation, the keys and values for prior tokens are cached and reused, turning what would be quadratic work per token into linear.
• Paged attention. vLLM-style memory management for the KV-cache, so many concurrent requests share GPU memory efficiently.
• FlashAttention (and FlashAttention-2/3). Fused attention kernels that avoid materialising the full attention matrix in HBM, dramatically reducing memory bandwidth pressure.
• Speculative decoding. A small draft model proposes several tokens; the large model verifies them in parallel. Latency drops without changing the output distribution.
• Quantisation and MoE routing. INT8/INT4 weights, mixture-of-experts gating, and similar tricks shrink the cost per token at serving time.

None of this stack exists for RNNs at the same maturity. Even if a new sequence architecture beat the transformer on a benchmark tomorrow, it would take years to rebuild the serving infrastructure. This is part of why the transformer’s dominance is sticky in a way that is not purely about the architecture itself.

Limitations and where the architecture is being modified

Transformers are not the final answer, and treating them as such is one of the failure modes we watch for during feasibility audits.

• Quadratic attention cost. Long contexts get expensive fast. Sparse attention, linear attention, sliding-window attention, and state-space hybrids (Mamba, Jamba) all attack this in different ways.
• Data hunger. Transformers tend to need more data than CNNs to generalise on vision tasks of comparable size. For small-data problems, the inductive bias of a CNN or a classical model can still be the right answer.
• Interpretability. Attention maps are suggestive but not faithful explanations of model behaviour. Mechanistic interpretability research is making progress, but transformer internals remain harder to audit than, say, a gradient-boosted tree.
• Infrastructure cost. The largest transformer models carry serious compute, memory, and energy footprints. For most production problems this is the dominant constraint, not the model architecture.

The 2026 research direction is not “replace the transformer” so much as “hybridise it”. Mixture-of-experts variants reduce active parameter count per token. State-space layers handle very long contexts where attention would be prohibitive. Diffusion-transformer hybrids power image and video generation. The attention block stays; the surrounding machinery shifts.

Frequently asked questions

Why did symbolic AI fail in the way it did, and what does neuro-symbolic AI bring back?

Symbolic AI failed because hand-coded rules and knowledge bases could not keep up with the combinatorial complexity of real-world perception, language, and reasoning. Statistical learning, and later deep learning, solved the perception and pattern-matching half. Neuro-symbolic AI brings back the explicit-reasoning half on top of learned representations — using transformer-based encoders to ground symbols in data, and symbolic layers to enforce constraints, verify outputs, and reason compositionally where pure neural networks remain brittle.

How does a working taxonomy of ML, deep learning, LLMs, and GenAI map to real engineering decisions?

Each level constrains data, compute, and failure modes differently. Classical ML wants tabular data and engineered features; failures show up as bias or distribution shift. Deep learning wants large labelled datasets and accelerators; failures show up as overfitting and adversarial fragility. LLMs and generative AI want enormous unlabelled corpora and serving infrastructure; failures show up as hallucination, prompt sensitivity, and cost-per-token. A team that names the family up front avoids the late-stage discovery that the problem was never a generative one.

What is the key feature of generative AI that separates it from classical ML for a production team?

Generative AI produces structured output — text, images, audio, code — sampled from a learned distribution, rather than predicting a single label or value. For a production team this changes evaluation (no single ground truth), latency (autoregressive generation costs scale with output length), and risk (outputs may be fluent and wrong). Classical ML is a function approximator; generative AI is a sampler. The engineering practices around the two are not interchangeable.

Where do transformers sit in the taxonomy, and why do they keep dominating across modalities?

A transformer is a deep-learning architecture, not a category of system. It shows up in discriminative models (BERT, ViT), generative models (GPT-class LLMs, diffusion transformers), and multi-modal systems (CLIP, Whisper, Gemini-class). It dominates because self-attention scales cleanly with data and compute, transfers across modalities with minimal redesign, and sits on top of a serving stack (KV-cache, paged attention, FlashAttention, speculative decoding) that has been hardware-optimised for it specifically.

How does applied AI differ from general AI in terms of what an engineering team should build today?

Applied AI solves a specific, bounded problem with measurable success criteria — a translation system, a defect-detection model, a domain-specific assistant. General AI (AGI) refers to systems that match human flexibility across arbitrary tasks, which remains a research aspiration, not an engineering target. Production teams in 2026 should build applied AI: pick the narrow problem, choose the smallest architecture that solves it, and reserve general-purpose models for genuine multi-task or long-tail scenarios.

Which technologies have actually advanced LLM operation in the last 24 months, and which are noise?

Real advances: FlashAttention-2/3, paged attention (vLLM), speculative decoding, mixture-of-experts routing, INT4/INT8 quantisation, and the maturing toolchain around retrieval-augmented generation. State-space hybrids (Mamba, Jamba) are early but credible for long-context regimes. Largely noise: claims of “reasoning” breakthroughs that turn out to be prompting tricks, marketing around context-window length without a serving-cost story, and most “agent framework” launches that wrap the same APIs with no new mechanism.

Where this connects in our work

Transformer architecture decisions show up early in any GenAI feasibility audit: which model family fits the problem, what the serving cost will look like at the target throughput, and whether a smaller non-transformer architecture would do the job at a fraction of the infrastructure budget. We have seen plenty of projects where the right answer was a fine-tuned encoder transformer; we have seen others where a classical model with engineered features would outperform a frontier LLM at one percent of the cost. The architecture is a tool, not a default.

If you are scoping a system and the conversation keeps blurring “transformer”, “LLM”, “deep learning”, and “generative AI”, the underlying taxonomy is worth getting right before any vendor selection happens. We cover the full picture in our companion piece on symbolic, generative, and traditional ML as a working taxonomy, and the broader programme context lives in our Generative & Agentic AI R&D practice.

Image credits: Freepik