Ablation Study — what does each piece of the model actually do?
An ablation study answers the question: 'If I take this part out, does the model get worse?' It is the cleanest way to prove which design choices matter. This lecture walks through ours, chart by chart, in plain English.
What is an ablation study?
Plain English first.
Imagine you bake a cake with flour, sugar, eggs, butter, and vanilla. You want to know which ingredient matters most. You bake five cakes: the full recipe, and four variants — each one with exactly one ingredient removed. You taste them and rank the results. The ingredient whose removal ruins the cake the most is the most important.
An ablation study in deep learning is exactly the same idea: train the model many times, each time removing or replacing one component (a skip connection, the focal loss, the Siamese pre-image branch…). Compare scores. The biggest drop reveals the most important component. The word "ablation" literally means "removal".
The experimental setup
Same data, same training, only one thing changes per run.
| Setting | Value | Why fixed |
|---|---|---|
| Dataset | xBD (train / val / hold-out test) | Identical evaluation across all runs |
| Image size | 1024 × 1024 patches → 224 × 224 | Match the encoder's pre-trained input |
| Optimiser | AdamW, lr = 1e-4, cosine schedule | Same for every variant — fair comparison |
| Epochs | 60 with early stopping | Long enough to converge, short enough to repeat |
| Loss (baseline) | Focal(γ=2) + Dice + class weights | Each ablation removes ONE term |
| Random seed | Fixed (42) | Reproducible — different runs are not just noise |
| Metric | Macro-F1 over 4 damage classes | Treats rare classes equally to the common one |
The golden rule. Only one thing changes between two runs. If you change two things at once (e.g. remove skip and drop focal loss), you can no longer tell which change caused the drop.
Figure A — Which architecture wins overall?
Macro-F1 on the held-out test set.
How to read it. Four bars, four architectures. The taller the bar, the better the model.
- ResNet-50 U-Net (0.62) — a strong CNN baseline. Good at local texture, weaker on global context.
- ViT-B/16 U-Net (0.66) — replacing the CNN encoder with a Vision Transformer adds long-range attention; +0.04 over ResNet.
- Siamese ResNet-50 (0.71) — feeding pre- AND post-disaster images through twin ResNets gives the network a "before vs after" view; +0.09 over the same encoder alone.
- Siamese ViT-B/16 (0.74) — the best of both: global attention + change detection. Winner.
Take-away. Both ideas help independently and they stack — Siamese alone adds ~0.09, ViT alone adds ~0.04, doing both adds 0.12 (= 0.74 − 0.62).
Figure B — Where do the gains actually come from?
Per-class F1 — the average can hide the rare classes.
How to read it. Each cluster of 4 bars is one damage class. The 4 colours are the 4 architectures.
- No-damage bars are all tall (~0.86–0.91). Every model handles the easy class well.
- Minor damage is by far the hardest (0.31 → 0.51). Going from ResNet-only to Siamese ViT improves it by +0.20 — a 65% relative gain.
- Destroyed bars are high (0.69 → 0.84). Visually obvious in satellite imagery, so easier even for weaker models.
Figure C — Component ablation: pull out one piece, see what breaks
Holding the architecture fixed (Siamese ViT), what does each component contribute?
How to read it. The top green bar is the full baseline. Every bar below it has one component removed; the number on the right (Δ) shows how much Macro-F1 fell.
| Removed component | F1 | Δ | Plain-English interpretation |
|---|---|---|---|
| — (baseline) | 0.74 | — | Everything in place. |
| Siamese (single image) | 0.52 | −0.22 | Biggest drop. Comparing pre vs post is the single most important idea — without it the network has to guess what was there before. |
| Focal loss → plain CE | 0.55 | −0.19 | Almost as bad. Without focal, easy background pixels dominate the loss and rare damage classes get ignored. |
| Class weights | 0.59 | −0.15 | Removes the per-class rebalancing — Minor damage collapses to near-zero recall. |
| Skip connections | 0.61 | −0.13 | Decoder can no longer recover sharp building boundaries → blurry masks. |
| Dice loss | 0.68 | −0.06 | Smaller drop. Dice mainly improves boundary quality; CE+focal already gets the classification right. |
| Deep supervision | 0.70 | −0.04 | Smallest drop — speeds up training and helps a little, but the model still works without it. |
What you should take away
Three lessons from the ablation.
- Ranking importance. For this problem the importance order is: Siamese > Focal > Class weights > Skip connections > Dice > Deep supervision. Knowing this order tells future engineers which knobs to protect when they simplify the model for deployment.
- Look at per-class metrics. Macro-F1 is a single number that already weighs classes equally, but the per-class chart (Figure B) shows where the improvement happens. Always include per-class results when the dataset is imbalanced.
- Change exactly one thing at a time. If you remove both focal and Dice in the same run and F1 drops 0.20, you cannot say which loss caused it. Ablation tables are only trustworthy when each row differs from the baseline in exactly one cell.
- Macro-F1 — compute F1 for each class separately, then average. Treats rare classes the same as common ones.
- Δ (delta) — the change vs. the baseline. Negative means worse.
- Baseline — the full model you are comparing against. Picking a meaningful baseline is half of a good ablation.
- Controlled experiment — fix everything except the one variable you are testing.
- Ablation
- — remove ONE component and re-train; the drop measures that component's importance.
- Baseline
- — the full model every variant is compared against.
- Δ (delta)
- — change vs. baseline. Negative = worse.
- F1
- — harmonic mean of precision & recall;
2PR/(P+R). - Macro-F1
- — F1 computed per class, then averaged equally. Treats rare classes fairly.
- Precision
- — of predicted damaged pixels, how many were actually damaged.
- Recall
- — of truly damaged pixels, how many we caught.
- IoU
- — Intersection-over-Union; another overlap score (close cousin of Dice).
- Hold-out test set
- — data the model never saw during training or tuning.
- Seed
- — fixed random number so the experiment is reproducible.
- Controlled experiment
- — change exactly one variable, hold everything else fixed.
- CE
- — cross-entropy loss (the "plain" baseline we compare focal to).