EY Open Science Data Challenge 2024
Storm Damage Assessment from Satellite Imagery
EY Open Science Data Challenge 2024 · solo entry · presented at IEEE IGARSS 2024
I built a model that finds every building in satellite images taken after a hurricane and says whether it is damaged. Most of the work was choosing and annotating the right training images.
- Out of 11,000 entrants worldwide
- 2nd runner-up
- Out of 11,000 entrants worldwide
- Building annotations in the training and validation sets
- 4,000+
- Building annotations in the training and validation sets
- Validation mAP after 8 epochs of fine-tuning
- 0.50
- Validation mAP after 8 epochs of fine-tuning
01 · The problem
After a hurricane, find the damaged buildings fast
After Hurricane Maria hit Puerto Rico in September 2017, responders needed to know where the damage was. Satellite images exist, but they are usually inspected by hand. The challenge: locate each building and classify it into one of four classes.
- Undamaged residential
- Damaged residential
- Undamaged commercial
- Damaged commercial
Hover or tap a building to see what the model predicts.
02 · The dataset
Choosing which patches are worth labeling
The challenge came with satellite images but no labels. Building a relevant training set was the hardest part, and the most important one.
aOverlapping patches keep buildings whole
The large satellite image is cut into 512 × 512 pixel patches, each covering about 170 × 170 m. Halving the stride from 512 to 256 pixels gives more images and avoids buildings cut at a border.
3 patches. The building sits on a border. Neither patch sees it whole.
bLabel the patches that look like the test
Not every patch is worth labeling, and finding the relevant ones by eye is slow. I described every patch with 18 simple features and kept the patches closest to the validation images.
14 of 60 candidate patches are worth labeling. Hover or tap a validation image.
9 color features
Color histograms: what the patch is made of.
9 texture features
Local binary patterns: how fine or coarse it looks.
Nearest neighbors
Keep the training patches closest to each validation image.
03 · Annotation
Let a small model do the first pass
Drawing boxes from scratch is slow. Correcting boxes that are mostly right is fast. So a small model did the first pass, and I fixed its mistakes.
- 1
Label. Annotate a first small batch of patches by hand.
- 2
Train YOLO. Train a small detector on what is labeled so far.
- 3
Pre-label. Run it on new patches to get draft boxes and classes.
- 4
Correct. Fix the drafts by hand and add them to the dataset.
Before and after images
Only post-storm images are used for training. But cutting the pre-storm images into the same patches let me compare both and spot what changed.
Building footprints
Outlines from Microsoft's Global ML Building Footprints gave a second reference for where the buildings are.
04 · The model
Fine-tuning Co-DETR
Damage assessment does not need real-time speed, so I picked the most accurate detector available over the fastest. Co-DETR, published in 2023, is an improved DETR-style transformer.
01
Co-DETR
Pretrained on COCO: 200,000+ labeled images, 80 categories
02
Fine-tune
8 epochs on my Puerto Rico patches, with crops, translations and contrast changes
03
Prediction
For each building: a box, one of 4 classes and a confidence score
- Imagery: Maxar GEO-1, Puerto Rico after Hurricane Maria
- Train, validation and test splits
- Checkpoint with the best validation score kept
05 · Results
Good at finding buildings, weaker on rare classes
The model reached a mAP of 0.60 on the training set and 0.50 on validation. It is strong on the common class and struggles on the rare ones.
| Truth ↓ · Predicted → | Undamaged res. | Undamaged com. | Damaged res. | Damaged com. | Background |
|---|---|---|---|---|---|
| Undamaged res. | 90% | 0% | 1% | 0% | 7% |
| Undamaged com. | 35% | 46% | 0% | 0% | 17% |
| Damaged res. | 28% | 0% | 49% | 0% | 22% |
| Damaged com. | 31% | 18% | 6% | 0% | 43% |
| Background | 84% | 5% | 9% | 0% | 0% |
Hover or tap a row to read it. Outlined cells are correct predictions.
- Undamaged residential2,857+ 496 val.
- Damaged residential366+ 49 val.
- Undamaged commercial200+ 50 val.
- Damaged commercial20+ 7 val.
The errors follow the data. Undamaged residential buildings make up most of the training set, and damaged commercial ones appear only 20 times.
06 · Takeaways
What I learned
01
The dataset is the model
Patch selection and annotation took most of the effort and decided most of the result. The model was fine-tuned with the standard Co-DETR recipe.
02
Draft labels cut both ways
Correcting a model's boxes is much faster than drawing them. It also takes less attention, so some of its mistakes slip into the labels.
03
More labels did not help
I labeled more rare-class examples to fix the imbalance, and the test score dropped. Telling the four classes apart from above is hard, even for the annotator.
Publication
E. Deng, "Coastline Resilience: Leveraging Satellite Imagery and the Co-Detr Model for Storm Damage Assessment," IGARSS 2024 - 2024 IEEE International Geoscience and Remote Sensing Symposium, Athens, Greece, 2024, pp. 609-613.
Read the paper on IEEE Xplore