← Curriculum index·L0 · Project Overview · xBD + ATC-20 L1 · Vision Transformers L2 · Focal Loss, Dice Loss & Class Weights L3 · Siamese Neural Networks L4 · Encoder · ResNet-50 L5 · Decoder · U-Net L6 · Ablation Study L7 · Hyper-parameters

L7 · Tuning

Hyper-parameters: the knobs you turn before training

Learning rate, batch size, optimiser, weight decay, dropout, schedulers, and seeds — what they do, why they matter, and the exact values used for the xBD damage-assessment project.

In plain English — read this first

A neural network has two kinds of numbers: parameters (the millions of weights inside the model, learned automatically by gradient descent) and hyper-parameters (a handful of numbers you pick before training starts). Think of it like baking a cake: the recipe ingredients are parameters that get mixed together, but the oven temperature, the baking time, and the pan size are hyper-parameters — you choose them, and a bad choice ruins the cake no matter how good the ingredients are.

This lecture walks through every hyper-parameter we use in the xBD project, explains what each one actually does, shows the equation behind it, and lists the value we chose with the reasoning. By the end you should be able to defend each number on the hyper-parameter table in your project write-up.

§1

Parameters vs. hyper-parameters

The single most important distinction.

A model's parameters are the weights and biases that gradient descent updates every batch. A ResNet-50 has roughly 25 million of them. We never set these by hand. Hyper-parameters are the small set of decisions that govern how the parameters are learned — the learning rate, the optimiser, the batch size, how long to train, how much to regularise, and so on. Figure 1 makes the boundary explicit.

Figure 1 — Parameters live inside the model and are learned automatically by the optimiser. Hyper-parameters live outside the model and are chosen by you. A bad set of hyper-parameters cannot be fixed by more training.

Tip

Rule of thumb. If a number is updated by loss.backward() it is a parameter. If you set it once at the top of your training script, it is a hyper-parameter.

§2

The learning rate — η

The single most impactful hyper-parameter in deep learning.

Gradient descent updates each weight by walking downhill on the loss surface. The learning rate η controls how big each step is:

\theta_{t+1} \;=\; \theta_{t} \;-\; \eta\,\nabla_{\theta}\,\mathcal{L}(\theta_t)

where

$\theta_t$ — weights at step t
$\eta$ — learning rate — the step size
$\nabla_{\theta}\mathcal{L}$ — gradient of the loss with respect to the weights — points uphill

Vanilla SGD: take a small step in the direction that most decreases the loss. The minus sign flips uphill into downhill.

Too small and the network crawls; too large and it overshoots the minimum and diverges. Figure 2 shows the three regimes on a simple bowl-shaped loss.

Figure 2 — Each dot is one optimiser step. Yellow (small η) barely moves; green (just-right η) descends quickly into the bowl; red (large η) bounces past the minimum.

In practice we never use a single fixed $\eta$ for the whole run. We warm up (start tiny, ramp up linearly for the first few epochs to avoid blowing up randomly-initialised weights), then anneal (smoothly shrink $\eta$ so the network can settle into a narrow minimum at the end). Figure 3 plots the three schedulers you will see in the wild.

Figure 3 — Three popular learning-rate schedules over 100 epochs. The xBD project uses warm-up + cosine (green dashed): a 10-epoch linear ramp avoids early instability, then a half-cosine smoothly anneals η down to ~5% of the peak so the network can settle into a sharp minimum.

For the xBD project we use warm-up + cosine annealing over 50 epochs:

\eta(e) \;=\; \begin{cases} \eta_{\max}\,\dfrac{e}{E_w}, & e < E_w \\[10pt] \eta_{\min} + \tfrac{1}{2}\bigl(\eta_{\max} - \eta_{\min}\bigr)\!\left[1 + \cos\!\left(\pi\,\dfrac{e - E_w}{E - E_w}\right)\right], & e \ge E_w \end{cases}

where

$e$ — current epoch (0, 1, …, E−1)
$E_w$ — number of warm-up epochs (we use 5)
$E$ — total epochs (we use 50)
$\eta_{\max}$ — peak learning rate at the end of warm-up (3 × 10⁻⁴)
$\eta_{\min}$ — floor learning rate at the end of training (1 × 10⁻⁶)

Two-phase schedule: linear warm-up for the first E_w epochs, then half-period cosine decay down to η_min. The cosine shape gives a long, gentle tail at the end so the optimiser can settle into a narrow minimum.

§3

Batch size — B

Statistical noise vs. memory vs. speed.

The batch size is how many training examples we average gradients over before taking one optimiser step. Small batches give noisy gradients that act as a mild regulariser; large batches give precise gradients but need a proportionally larger learning rate to compensate.

\eta_{\text{new}} \;=\; \eta_{\text{base}}\cdot\dfrac{B_{\text{new}}}{B_{\text{base}}}

where

$B$ — batch size
$\eta_{\text{base}}$ — learning rate that already works for B_{base}

Linear scaling rule (Goyal et al., 2017): doubling the batch size doubles the effective signal-to-noise of each gradient, so we can take a step twice as big without diverging.

Batch size	GPU VRAM (≈)	Steps per epoch	Pros	Cons
4	~6 GB	~22 k	Fits any GPU	Very noisy gradients
16	~12 GB	~5.5 k	Good noise/precision balance	Slower per epoch
32	~22 GB	~2.7 k	Project default	Needs a 24 GB GPU
64	~44 GB	~1.4 k	Faster convergence	Needs A100, may need LR re-tune

Table 1 — Batch-size trade-off for the ResNet-50 + U-Net segmentation network on 512×512 xBD tiles.

§4

Optimisers

SGD, Adam, AdamW — and why we use AdamW.

The optimiser is the algorithm that uses the gradient to update the weights. Plain SGD uses the gradient directly. Adam adapts a per-weight step size based on the running mean and running variance of the gradient — much faster on noisy problems like ours.

\begin{aligned} m_t &= \beta_1\,m_{t-1} + (1-\beta_1)\,g_t \\[4pt] v_t &= \beta_2\,v_{t-1} + (1-\beta_2)\,g_t^{\,2} \\[4pt] \theta_{t+1} &= \theta_t \;-\; \eta\,\dfrac{\hat{m}_t}{\sqrt{\hat{v}_t} + \varepsilon} \end{aligned}

where

$g_t$ — current gradient \nabla_{\theta}\mathcal{L}
$m_t$ — running average of the gradient (momentum, β₁ = 0.9)
$v_t$ — running average of the squared gradient (β₂ = 0.999)
$\hat{m}_t,\;\hat{v}_t$ — bias-corrected versions of m and v
$\varepsilon$ — small constant (10⁻⁸) to avoid division by zero

Adam: keep two exponential moving averages — one of the gradient (m, the momentum) and one of its square (v, the variance). The per-weight effective step size η · m̂ / √v̂ shrinks for noisy weights and grows for stable ones.

AdamW is Adam with the weight-decay term decoupled from the gradient update (Loshchilov & Hutter, 2017). Standard Adam mixes weight decay into the running averages, which under-penalises large weights. AdamW fixes this and is the modern default for both ViT and CNN training.

Tip

Project choice. AdamW with

\eta_{\max} = 3 \times 10^{-4}

\beta = (0.9,\,0.999)

, weight decay

\lambda = 1 \times 10^{-4}

. These are the defaults from the original ViT paper and they transfer well to our hybrid encoder.

§5

Weight decay & dropout

Two regularisers that fight overfitting.

Overfitting happens when the network memorises the training set instead of learning patterns that generalise to new tiles. Figure 4 is the classic shape: train loss keeps dropping while validation loss bottoms out and then climbs.

Figure 4 — Train loss (green) keeps falling because the network is memorising the training tiles. Validation loss (red) bottoms out around epoch 25 and then climbs — that's overfitting. The dashed line marks where to early-stop; weight decay, dropout, and stronger augmentation push that point further to the right.

We push the validation curve down (and rightward) with two cheap regularisers:

\mathcal{L}_{\text{total}} \;=\; \mathcal{L}_{\text{data}}(\theta) \;+\; \lambda\,\lVert\theta\rVert_{2}^{2}, \qquad \lambda = 10^{-4}

where

$\mathcal{L}_{\text{data}}$ — the usual data loss (Focal + Dice in our case)
$\lVert\theta\rVert_{2}^{2}$ — sum of squared weights — large weights are heavily penalised
$\lambda$ — weight-decay strength (small, but nonzero)

Weight decay (L2 regularisation): adds a tax on big weights. The optimiser now has to balance fitting the data against keeping the weights small, which prevents the network from memorising training noise.

\tilde{a}_i \;=\; a_i \cdot \dfrac{\mathrm{Bernoulli}(1-p)}{1-p}, \qquad p = 0.1\;\text{(decoder)},\; 0.0\;\text{(encoder)}

where

$a_i$ — activation of neuron i in the layer
$p$ — drop probability — fraction of neurons to silence each step
$\mathrm{Bernoulli}(1-p)$ — 1 with probability 1-p, otherwise 0 — the random mask
$1/(1-p)$ — rescale so the average activation stays the same

Dropout (training only): randomly zero a fraction p of activations on each forward pass. The network cannot rely on any single neuron, so it learns redundant, more robust features.

§6

Other knobs you must set

Hyper-parameter	What it controls	Project value	Why
Epochs E	How many full passes through the data	50	Validation F1 plateaus around epoch 42 — leave 8-epoch safety margin
Warm-up epochs E_w	Length of the linear LR ramp	5	Standard for AdamW with batch size 32
Loss weights (Focal, Dice)	Relative pull of each loss term	α=1.0, β=1.0	Equal balance worked best in our pilot ablation
Class weights w_c	Up-weight rare damage classes	[0.5, 1.0, 2.5, 5.0, 5.0]	Inverse √frequency from xBD train split
Augmentation strength	How aggressive the random flips/colour jitter are	Medium (Albumentations 'default')	Heavy aug hurt small-building recall in pilot
Random seed	Reproducibility of weights, shuffles, augmentations	1337	We re-run 3 seeds to report mean ± std
Gradient clipping	Cap on the global gradient norm	1.0	Stops occasional exploding-gradient spikes from ViT-like layers

Table 2 — Complete hyper-parameter set used for every ablation run in the project. Holding these fixed across architectures is what makes the comparison fair.

§7

The tuning workflow

How to find a good setting without burning a thousand GPU hours.

Figure 5 — A disciplined five-step tuning workflow. Skipping step 2 (sanity-overfit one batch) is the #1 source of wasted GPU hours in undergraduate projects: if the network can't memorise 8 tiles, no hyper-parameter setting will save it.

Start from published defaults. ResNet-50 + U-Net on segmentation? Use AdamW, η = 3e-4, B = 32. Don't invent.
Sanity-overfit one batch. Take 8 tiles, train for 200 steps, and confirm loss reaches near-zero. If it doesn't, the bug is in your code, not your hyper-parameters.
Run an LR finder. Sweep η from 10⁻⁶ to 10⁻¹ for ~100 steps and plot loss; pick the value one order of magnitude below the point where loss explodes.
Tune one knob at a time. Just like an ablation study — change one number, keep everything else fixed, measure the validation F1, decide.
Lock in and run the full 50 epochs with seeds and a logger (TensorBoard or W&B).

§8

Common failure modes

If your training looks like one of these, you have a hyper-parameter problem, not a model problem.

Symptom	Likely cause	Fix
Loss is NaN after a few steps	η too high or no gradient clipping	Halve η, set clip_grad_norm=1.0
Loss flatlines from step 1	η too low, or learning-rate not connected to optimiser	Run the LR finder; verify the scheduler is stepped each epoch
Train F1 great, val F1 terrible	Overfitting	Increase weight decay, dropout, augmentation; reduce epochs; early-stop
Different runs disagree by > 3 F1	Insufficient seed averaging	Run 3+ seeds, report mean ± std
First epoch loss spikes then recovers	No warm-up; AdamW state is cold	Add 3–5 warm-up epochs

Table 3 — A diagnostic cheat-sheet. Memorise this before your first long training run.

§9

Wrap-up & next lecture

Hyper-parameters are the unglamorous half of deep learning — but they decide whether the same architecture gets 0.55 or 0.71 macro-F1. The values in Table 2 are the ones we lock down across every ablation run so that any metric change is attributable to the component we changed, not to a lucky learning rate.

Tip

Checkpoint. You should be able to (1) explain the difference between parameters and hyper-parameters, (2) defend each value in Table 2, (3) describe what each scheduler shape in Figure 3 does, and (4) diagnose a NaN loss from Table 3.

← L6 · Ablation Study Curriculum index Take the quiz →

Acronyms & jargon — quick reference

η (eta) / LR: — learning rate; step size of each weight update.
B: — batch size; examples averaged per optimiser step.
E / E_w: — total epochs / warm-up epochs (linear ramp at start).
epoch: — one full pass over the training set.
SGD: — Stochastic Gradient Descent: the plain "take a step downhill" optimiser.
Adam / AdamW: — adaptive optimiser; AdamW decouples weight decay (modern default).
β₁, β₂: — Adam's momentum decay rates (0.9, 0.999 by default).
m_t, v_t: — Adam's running mean and variance of the gradient.
λ (lambda): — weight-decay strength; bigger λ → smaller weights → less overfitting.
L2 / ‖θ‖²: — sum of squared weights, the thing weight decay penalises.
Dropout p: — probability of zeroing a neuron during training (we use 0.1 in the decoder).
Cosine annealing: — half-cosine schedule that smoothly drops η from η_max → η_min.
Warm-up: — linear ramp from 0 → η_max during the first E_w epochs.
Grad clipping: — cap the gradient's norm to a fixed value (we use 1.0) to stop explosions.
Early stop: — stop training once validation loss stops improving.
VRAM: — GPU memory. Bigger B → more VRAM.
Seed: — fixed random number so a run is reproducible.
F1 / macro-F1: — F1 averaged equally over all classes (rare classes count the same as common).