← Curriculum index·L0 · Project Overview · xBD + ATC-20 L1 · Vision Transformers L2 · Focal Loss, Dice Loss & Class Weights L3 · Siamese Neural Networks L4 · Encoder · ResNet-50 L5 · Decoder · U-Net L6 · Ablation Study L7 · Hyper-parameters

Lecture L1

Vision Transformers (ViT)

Architecture, attention mechanism, variants, and the vanishing-gradient story — a complete undergraduate lecture adapted from the CEAMLS slide deck.

Patch embeddingSelf-attentionMulti-Head (MSA)Encoder blockLayerNorm + ResidualsDeiT · Swin · BEiT · CaiT · MaxViTVanishing gradient proofViT vs CNN

In plain English — read this first

What is a Vision Transformer? A regular CNN looks at an image through small sliding windows and slowly builds up what it sees. A Vision Transformer (ViT) instead chops the image into a grid of small square patches (think of cutting a photo into 16×16-pixel postage stamps), turns each patch into a vector, and lets every patch talk to every other patch from the very first layer. That "talking" is called self-attention.

Why is that useful? A CNN needs many layers before two pixels on opposite sides of the image can influence each other. A ViT can connect them in a single step. For satellite imagery — where a damaged roof on one corner of the tile relates to debris on the other corner — this long-range view is a big advantage.

Vision Transformer (ViT) architecture: input image split into 16×16 patches, linear projection, patch + positional embeddings with [CLS] token, transformer encoder block (LayerNorm, Multi-Head Self-Attention, MLP, residuals) repeated L×, then MLP head for class label. — **Figure 1.** End-to-end ViT pipeline: patchify → linearly project → add positional embeddings → L stacked Transformer encoder blocks → MLP head.

Figure walkthrough — read the diagram block by block

① Input Image (H×W×3) — one RGB photo, e.g. a 224×224 satellite tile of a building. The 3 is the red/green/blue channels.
② Split into Patches (16×16) — chop the image into a grid of small square "tiles". A 224×224 image gives 14×14 = 196 patches. Think of each patch as a "word" in a sentence.
③ Linear Projection (Flatten + Linear) — each patch (16·16·3 = 768 numbers) is multiplied by a learnable matrix E. This turns a raw patch into a vector the Transformer can understand. (It is mathematically identical to a single Conv 16×16, stride 16.)
④ Patch Embeddings — the stack of vectors that comes out of step 3, one per patch. The extra purple [CLS] token at the top is a learnable "summary" vector — its final value is what we use to classify the image (BERT-style).
⑤ Positional Embeddings (+) — self-attention is order-blind, so we add a small learnable vector that says "I am patch #3 in row 1". Without this, the model could not tell top-left from bottom-right.
⑥ Transformer Encoder (L×) — the heart of ViT, repeated L times (e.g. 12 in ViT-Base). Inside each block: Layer Norm stabilises values → Multi-Head Self-Attention lets every patch look at every other patch → residual add (⊕) preserves the input → Layer Norm → MLP (a small feed-forward network refining each token) → another residual add.
⊕ Residual connections — the curved "skip" arrows that add the input back to the output. They are what stops the gradient from vanishing in deep stacks (see §7 below).
⑦ MLP Head — a tiny classifier (one or two dense layers) that reads only the final [CLS] vector.
⑧ Class Label — the prediction. In our damage-assessment context: "building / no-damage / minor / major / destroyed".
Legend symbols — D = embedding dimension (768 in ViT-Base), L = number of encoder blocks, N = number of patches.

Why Transformers for Vision? CNNs vs ViT

Fundamental limitations of convolution and what Transformers offer instead.

CNN limitation

A 3×3 kernel sees only 9 pixels at a time — a local receptive field. To let a top-left pixel influence a bottom-right pixel, you must stack many conv layers, and the global receptive field is still approximate and path-dependent.

What ViT brings

Global self-attention — every patch attends to every other patch in a single layer.
No baked-in inductive bias — the model learns translation invariance (or not) from data.
Unified architecture with NLP Transformers → multi-modal models, transfer learning.
Scales with data and parameters; ViT-G (6B params) hits 90%+ on ImageNet.

Feature	CNN	ViT
Context range	Local (kernel size)	Global (all patches)
Inductive bias	Strong (translation inv.)	Weak — learned from data
Data needed	Less (biases help)	More, or pre-training
Computation	O(N·k²)	O(N²) — quadratic in tokens
Long-range dependencies	Many stacked layers	One single layer ✓

Patch Embedding & Tokenisation

How an image becomes a sequence of tokens — the very first step of ViT.

Original image — a 224×224 pixel RGB photo. "RGB" = three colour channels (Red, Green, Blue), so the tensor shape is [3, 224, 224].
Extract patches — divide the image into non-overlapping P×P squares with P=16. That gives 14×14 = N=196 patches, and each patch holds 16·16·3 = 768 raw numbers. Think of each patch as one "word" in a 196-word sentence.
Linear projection (the "embedding" step) — flatten each patch into a length-768 row and multiply by a learnable matrix $E \in \mathbb{R}^{P^2 C \times D}$ . Out comes a D=768-dimensional vector per patch. (Trivia: this is mathematically identical to applying a single Conv 16×16 with stride 16 — convolution and patch embedding are the same op here.)
Prepend the [CLS] token — we glue an extra learnable vector to the front of the sequence. "CLS" = "classification". After L layers of attention, its final value summarises the whole image and is what we hand to the classifier. (Borrowed from BERT in NLP, where the same trick summarises a sentence.)
Add Positional Encoding (PE) — attention is "set-like": it doesn't know which patch came first. So we add a learnable vector for every position (1…197) that says "I'm patch #3 of row 1". Without PE, scrambling the patches would give the same answer.

[3, 224, 224] → 196 × [768] → linear → [196, 768] → + CLS → [197, 768] → + PE → [197, 768]

Vocabulary check: "token" = one row of the [197, 768] matrix. "Embedding dimension D" = 768 (the row length). "Sequence length" = 197 (number of rows = 196 patches + 1 CLS token).

Self-Attention — Query, Key, Value

The mathematical core of the Transformer — how patches 'talk to' each other.

Intuition before maths. Imagine a classroom of 197 students (= the 197 tokens). Each student writes three sticky-notes: a Question (Q — "I'm looking for information about…"), a Tag (K — "I know about…"), and a Note (V — the actual content they will share). Self-attention is the rule: every student compares their Q to everyone else's K, and copies a weighted average of everyone's V — listening hardest to whoever matches them best.

In maths, each token's Q, K, V is built by multiplying the token's vector $X$ by three learned matrices: $Q = X W_Q$ , $K = X W_K$ , $V = X W_V$ .

Scaled dot-product attention

\mathrm{Attention}(Q,K,V) \;=\; \mathrm{softmax}\!\left(\frac{Q\,K^{\top}}{\sqrt{d_k}}\right) V

where

$Q = X W_Q$ — queries — what each token is looking for
$K = X W_K$ — keys — what each token offers
$V = X W_V$ — values — the content each token will share
$d_k$ — dimension of each key vector (64 in ViT-Base)
$Q K^{\top}$ — [N×N] similarity matrix — entry (i,j) = how much token i wants token j
$\mathrm{softmax}$ — row-wise: turns each row of scores into probabilities that sum to 1

Step by step: (1) compare every token to every other token via $Q K^{\top}$ (one big dot-product matrix of similarities); (2) divide by $\sqrt{d_k}$ so the numbers don't blow up; (3) softmax each row so the weights for one token sum to 1; (4) multiply by $V$ to take a weighted average of values. Each token's new representation is a mixture of every other token, weighted by relevance.

Why divide by √d_k?

For large

d_k

, raw dot products grow large → softmax saturates (one entry ≈ 1, the rest ≈ 0) → gradient ≈ 0 (a vanishing-gradient problem on the attention path). Scaling by

\sqrt{d_k}

keeps the logits in a healthy range. With

d_k = 64

we divide by 8; with

d_k = 256

we divide by 16.

Multi-Head Self-Attention (MSA)

Running attention in parallel across multiple representation subspaces.

Multi-head self-attention

\begin{aligned} \mathrm{head}_i &= \mathrm{Attention}\!\bigl(X W_Q^{(i)},\;X W_K^{(i)},\;X W_V^{(i)}\bigr) \\[4pt] \mathrm{MSA}(X) &= \mathrm{Concat}\!\bigl(\mathrm{head}_1,\dots,\mathrm{head}_h\bigr)\,W_O \end{aligned}

where

$h$ — number of attention heads (12 in ViT-Base)
$W_Q^{(i)}, W_K^{(i)}, W_V^{(i)}$ — per-head projection matrices — each head sees a different 64-D subspace
$W_O$ — output projection — mixes the h head outputs back into D = 768

One attention layer can only learn one kind of relationship. Running $h$ heads in parallel lets the model attend to different things simultaneously — e.g. head 1 on nearby texture, head 2 on long-range objects, head 3 on object boundaries. Concatenating their outputs and projecting with $W_O$ gives a single richer representation.

Different aspects — Head 1 may focus on texture (nearby patches), Head 2 on semantic similarity, Head 3 on object boundaries.

Richer representations — h heads project into h subspaces in parallel — far richer than one.

ViT-Base — h=12 heads, D=768, d_k=d_v=64. 12×64=768 (dims preserved). ~2.36M params per MSA layer.

Complete ViT Architecture

From image → patches → 12 encoder blocks → CLS → classification head.

Image 224×224×3
   ↓ Patch Embed (16×16, stride 16)
   ↓ + [CLS] + Positional Encoding
[197 × 768]
   ↓ × 12 Encoder Blocks
[197 × 768]
   ↓ extract CLS token
[1 × 768]
   ↓ MLP head
[K classes]

ViT-Base config: P=16, N=196, D=768, L=12 blocks, h=12 heads, d_k=64, FFN hidden=3072 (4·D), 86M parameters.

Variant	Layers	Hidden D	Heads	Params
ViT-Small	12	384	6	22M
ViT-Base	12	768	12	86M
ViT-Large	24	1024	16	307M
ViT-Huge	32	1280	16	632M

Transformer Encoder Block — Inside One Block

Two residual sub-blocks: Pre-LN → MSA → ⊕, Pre-LN → FFN → ⊕.

Every term in this section, in plain English

The equations below mention six things. Here is what each one is and why it's there — read this first, then the maths will look obvious.

x — the token tensor. A matrix of shape [197 × 768]: one row per token. Everything inside the block reads and writes this matrix.
LN — Layer Normalisation. A tiny "auto-volume-knob" that, for each token's 768-number vector, subtracts the mean and divides by the standard deviation, then re-scales with learnable parameters γ and β. Why? Without it, the numbers drift larger and larger as we stack blocks, eventually exploding or saturating activations. LN keeps every token in a sane range. "Pre-LN" just means we apply LN before MSA/FFN (more stable than the original "Post-LN" order from the 2017 paper).
MSA — Multi-Head Self-Attention. The "patches talk to each other" step from §3 and §4, run with h=12 parallel heads. Output shape is the same as input ([197 × 768]) so we can add it back. Mixes information across tokens.
FFN — Feed-Forward Network. A small two-layer MLP applied independently to each token: expand 768→3072, apply GELU, project back 3072→768. Think of it as a per-token "refine my representation now that I know what my neighbours said". Does NOT mix tokens.
GELU — Gaussian Error Linear Unit. A smooth activation function. Where ReLU is a sharp corner at 0 ( $\max(0,x)$ ), GELU bends smoothly: it's roughly $x \cdot \Phi(x)$ where $\Phi$ is the Gaussian cumulative. Smoother → better gradient flow → trains slightly better than ReLU in Transformers.
+ x (residual / skip connection). The output of each sub-block is added back to its input. This is the "gradient highway" — see §7 for the proof. Visually, in the architecture figure, these are the curved arrows that bypass the sub-block.

Sub-block 1 (token mixer)

$x' = \mathrm{MSA}(\mathrm{LN}(x)) + x$ — normalise, let tokens attend to each other, then add the original back.

Sub-block 2 (token refiner)

$x'' = \mathrm{FFN}(\mathrm{LN}(x')) + x'$ — normalise again, refine each token independently, add the previous state back.

One-line summary: a block = "normalise, mix, add" then "normalise, refine, add". Repeat 12 times.

One Transformer block (Pre-LN)

\begin{aligned} x' &= \mathrm{MSA}\!\bigl(\mathrm{LN}(x)\bigr) + x \\[4pt] x'' &= \mathrm{FFN}\!\bigl(\mathrm{LN}(x')\bigr) + x' \end{aligned}

where

$\mathrm{LN}$ — layer normalisation (see below)
$\mathrm{MSA}$ — multi-head self-attention from §4
$\mathrm{FFN}$ — per-token feed-forward network (below)
$+\,x$ — residual / skip connection — the gradient highway

Each block has two residual sub-blocks: one for mixing tokens (MSA), one for transforming each token independently (FFN). The $+\,x$ on the end of each line is what lets gradients flow back through 12, 24, even 32 stacked blocks without dying.

Layer normalisation (per token, across D features)

\mathrm{LN}(x) \;=\; \gamma \,\odot\, \frac{x - \mu}{\sqrt{\sigma^2 + \varepsilon}} \;+\; \beta

where

$\mu, \sigma^2$ — mean and variance of the D = 768 features of that token
$\gamma, \beta$ — learnable per-feature scale and shift
$\varepsilon$ — small constant (e.g. 10^{-5}) for numerical stability

Centres and scales each token's features so the activations stay in a healthy range. Without this, deep stacks drift towards exploding or vanishing values.

Feed-forward network (per token, independent)

\mathrm{FFN}(x) \;=\; \mathrm{GELU}\!\bigl(x W_1 + b_1\bigr)\,W_2 + b_2

where

$W_1 \in \mathbb{R}^{D \times 4D}$ — expansion from 768 → 3072
$W_2 \in \mathbb{R}^{4D \times D}$ — projection back 3072 → 768
$\mathrm{GELU}$ — Gaussian Error Linear Unit — a smooth ReLU

The FFN acts on each token independently (no cross-token mixing). Think of it as a small per-token MLP that decides "given what this token now represents, transform it like so". MSA mixes tokens; FFN refines them.

Pre-LN (normalise before MSA/FFN) is more stable than Post-LN for deep stacks.
MSA mixes tokens; FFN does not — FFN acts independently per token, a learned "memory" lookup.
GELU (Gaussian Error Linear Unit) smoothly gates the input; outperforms ReLU in Transformers.
Two residuals per block × 12 blocks = 24 gradient highways from output back to input.

Vanishing Gradient — Why Residuals Save ViT

A short proof that the +x in x' = F(x)+x guarantees gradient flow.

Residual block & its gradient

\begin{aligned} x_{\ell+1} &= F_{\ell}(x_{\ell}) + x_{\ell} \\[6pt] \frac{\partial \mathcal{L}}{\partial x_0} &= \frac{\partial \mathcal{L}}{\partial x_L}\;\prod_{\ell=0}^{L-1}\!\left(\frac{\partial F_{\ell}}{\partial x_{\ell}} + I\right) \end{aligned}

where

$x_{\ell}$ — activations entering block \ell
$F_{\ell}$ — the block's non-linear transform (MSA or FFN here)
$I$ — identity matrix from the skip connection
$L$ — total number of blocks (12 for ViT-Base)

Even if $\partial F_{\ell}/\partial x_{\ell} \to 0$ (the classic vanishing-gradient problem), the $+\,I$ term ensures every factor in the product is at least the identity. So the gradient can never collapse to zero before reaching the early layers — the skip connection is literally a gradient highway.

Combined with LayerNorm (keeps activations at μ=0, σ=1) and the $\sqrt{d_k}$ scaling inside attention, ViT trains stably to 12, 24, even 32 layers deep — something CNNs only achieved after the ResNet (2015) skip-connection breakthrough.

Numerical intuition

Suppose each

\partial F_\ell/\partial x_\ell

has typical magnitude 0.5. Without residuals, after L=12 layers the gradient is scaled by

0.5^{12} \approx 0.00024

— effectively zero, early layers stop learning. With residuals, each factor becomes

0.5 + I

(eigenvalues ≥ 1), and the product stays of order 1. That is why ViT can be 12+ layers deep without dying.

Jacobian (the $\partial F_\ell / \partial x_\ell$ term) = "how much the block's output wiggles when its input wiggles". A small Jacobian = the block barely passes information back. Adding the identity I forces it to pass something through, always.
ℒ (script L) is the loss function — the number we're minimising. Its gradient w.r.t. the early activations $x_0$ is what training needs.

ViT Variants

Five families you should know — when to reach for each.

Variant	Year	Key idea	Best for
DeiT	2021	Data-efficient ViT; distillation token from a CNN teacher.	Small/medium datasets — no JFT-300M pre-training needed.
Swin	2021	Shifted-window attention; hierarchical, linear complexity.	Dense prediction (detection, segmentation), high-res imagery.
BEiT	2021	BERT-style masked image modelling self-supervised pre-training.	Label-scarce domains (medical, satellite).
CaiT	2021	Class-Attention layers; LayerScale enables very deep ViTs.	Maximum ImageNet accuracy at fixed compute.
MaxViT	2022	Block-attention + grid-attention; hybrid with conv stem.	Long-range + local features in one network — strong all-rounder.

ViT vs CNN — When to Use Which

Situation	Prefer
< 50k labelled images, no pre-training	CNN (ResNet/EfficientNet)
Large pre-training corpus available (ImageNet-21k, JFT)	ViT
High-resolution dense prediction (segmentation)	Swin / MaxViT
Multi-modal (image + text)	ViT (shares Transformer with NLP)
Edge device, tight FLOPs budget	CNN or MobileViT
Satellite / medical, self-supervised pre-training	BEiT / MAE-pretrained ViT

Equations cheat sheet

text
1Patch embedding:   z_0 = [x_cls; x_p^1 E; x_p^2 E; … ; x_p^N E] + E_pos
2Attention head:    A_i = softmax(Q_i K_iᵀ / √d_k) V_i
3MSA:               MSA(x) = Concat(A_1,…,A_h) W_O
4Encoder block:     x' = MSA(LN(x)) + x ; z = FFN(LN(x')) + x'
5FFN:               FFN(x) = GELU(x W_1 + b_1) W_2 + b_2
6Classification:    ŷ = softmax( LN(z_L^{cls}) · W_head )

Key takeaway

ViT replaces convolution's local prior with global self-attention. With residual connections and LayerNorm, it scales gracefully to billions of parameters — and is the backbone of modern damage-assessment encoders in our Siamese pipeline.

Glossary — every acronym used in this lecture

Skim this once, then refer back whenever a symbol looks unfamiliar. Each term is also re-defined the first time it appears in the text.

ViT — Vision Transformer. The whole network in this lecture.

CNN — Convolutional Neural Network (ResNet, VGG, EfficientNet…). The "old" image model.

Patch — a small square chunk of the image (e.g. 16×16 px). Treated like a "word".

Token — a vector representing one patch (or the special [CLS] token).

Embedding — turning a patch (or word) into a vector of numbers the network can do math on.

[CLS] — "classification" token. A learnable extra vector whose final value is used to predict the class.

PE — Positional Encoding (or Embedding). Tells the model where each patch sits in the grid.

Q, K, V — Query, Key, Value vectors inside attention. Q = "what I want", K = "what I offer", V = "my content".

Attention — a weighted average where each token decides how much to listen to every other token.

Self-attention — attention where Q, K, V all come from the same sequence (the same image).

MSA — Multi-Head Self-Attention. Runs h attention "heads" in parallel, then concatenates.

Head — one independent attention computation inside MSA. ViT-Base has 12 heads.

FFN / MLP — Feed-Forward Network / Multi-Layer Perceptron. A tiny 2-layer fully-connected network applied per token.

LN — Layer Normalisation. Re-centres & re-scales a vector so its numbers stay healthy.

Pre-LN / Post-LN — whether LayerNorm is applied before the sub-block (modern, stable) or after (original, harder to train deep).

GELU — Gaussian Error Linear Unit. A smooth activation, smoother cousin of ReLU.

ReLU — Rectified Linear Unit,

\max(0,x)

. The classic activation.

Residual / Skip connection — a shortcut wire that adds a block's input to its output (the "⊕" in figures).

Softmax — turns a vector of raw scores into probabilities that sum to 1.

Logits — raw, un-normalised scores before softmax.

Encoder block — one Transformer unit: LN → MSA → ⊕ → LN → FFN → ⊕. Stacked L times.

L — number of encoder blocks (12 in ViT-Base, 24 in ViT-Large).

D — embedding dimension (length of each token vector). 768 in ViT-Base.

N — number of patches (e.g. 196 for a 224×224 image with 16×16 patches).

d_k — dimension of each attention head's key/query (64 in ViT-Base).

Inductive bias — built-in assumptions a model makes (e.g. CNNs assume nearby pixels matter most).

Vanishing gradient — when training signals shrink to ~0 in deep nets so early layers stop learning.

Jacobian — the matrix of all partial derivatives of a function's outputs w.r.t. its inputs.