Foundation model for SAR imagery

using MAE for pretraining a ViT backbone on SAR data

---
Highlights
-- Developed a SAR imagery foundation model using Masked Autoencoders (MAE) for feature learning.
-- Pre-trained Vision Transformer (ViT) on AI4Arctic dataset with MAE, fine-tuned for segmentation.
-- Achieved 5% F1-score increase over UNet benchmark with only 50% labeled data.
-- Model learned SAR features and irregular objects like ice, water, and land.
---
Masked Autoencoders
Schematics of MAE. Input patches are masked, and the deep learning model is asked to recover the masked patches.

I applied the MAE pre-training strategies and surprisingly, the regeneration results are quite impressive. Since our SAR signals are images of Earth's surface (Canadian Arctic in our case), I thought the deep learning model wouldn't be able to regenerate irregular objects like ice shapes, rocks, and water. But the results below prove otherwise:

MR_25
MR_75
MR_90
The above images show the regeneration quality with different mask ratios. Increasing the mask ratio leads to a loss of crucial information necessary for accurately reproducing features such as the crack lines (black lines), resulting in incorrect representations of their lengths.

Next, let's investigate the effect of pre-training on a downstream task (semantic segmentation of Sea Ice age).

Table MAE
Effect of Different Labelled Data
Figure (a) shows the effect of different pretraining strategies. Figure (b) shows the effect of % of pretraining vs fine-tuning data.

Lastly, let's visualize the segmentation quality using the pre-trained model.

Visualization of Segmentation
Segmentation visualization of SOD on two test scenes. The first row shows the Original HH, HV, and Ground truth, the second row shows the reconstructed HH, HV from a 75% masked image, and the third row shows the prediction of the AI4Arctic benchmark model. The land area is filtered out in the segmentation maps.