Graph Dynamic Earth Net: Spatio-Temporal Graph Benchmark for Satellite Image Time Series

Source code for the paper "Graph Dynamic Earth Net: Spatio-Temporal Graph Benchmark for Satellite Image Time Series" by Corentin Dufourg, Charlotte Pelletier, Stéphane May and Sébastien Lefèvre, at International Geoscience and Remote Sensing Symposium 2023 (IGARSS 2023).

We propose a comparison of five graph neural networks applied to spatio-temporal graphs built from satellite image time series.
The results highlight the efficiency of graph models in understanding the spatio-temporal context of regions, which might lead to a better classification compared to attribute-based methods.

	skip cnnct.	attnt.	Nb. of params	Impervious IoU	Agriculture IoU	Forest IoU	Wetlands IoU	Soil IoU	Water IoU	Test mIoU (↑)	Test Accuracy (↑)
RF	-	-	-	22.86±0.38	3.28±0.08	34.73±0.14	0.31±0.14	41.97±0.33	77.28±0.47	30.07±0.10	51.95±0.17
MLP	-	-	205k	19.51±3.11	0.85±0.56	41.86±1.59	0.43±0.71	48.92±1.60	80.75±2.24	32.05±0.46	58.33±1.37
GCN	✗	✗	205k	13.94±4.44	1.73±2.22	40.14±1.56	0.00±0.00	44.75±1.03	66.62±2.18	27.86±1.23	55.69±1.40
GSAGE	✓	✗	406k	28.36±4.21	4.86±2.38	47.27±3.04	0.86±0.55	51.97±2.15	83.46±1.51	36.13±1.33	63.01±2.06
GAT	✗	✓	207k	20.96±4.84	9.88±5.28	41.07±2.82	2.11±1.74	47.86±2.66	76.61±5.15	33.08±2.15	58.42±2.32
ResGGCN	✓	✓	811k	30.13±5.92	4.65±3.26	43.93±3.62	3.28±1.99	48.89±4.56	83.93±1.38	35.81±1.91	60.19±3.65

Table: IoU and Accuracy performance on the test set of attribute-based and GNN models. The results are provided with average and standard deviation on ten random initializations (best, second best).

Important

⚠️ Disclaimer on IGARSS paper results: The results provided here override those of the original paper by fixing a problem in graph creation.

Overview

• train_nn.py is the method to train the neural network models.
• train_rf.py is the method to reproduce the random forest results.
• config/ contains the configuration files to reproduce the paper results.
• data/ contains dataset and spatio-temporal graph classes.
• network/ contains the architectures implemented with PyTorch and PyG.
• datasets/ should contain the data from Dynamic Earth Net and the derived graphs to train and evaluate the models.
• helper/ contains the parser class for command-line options.
• utils/ contains utilitary methods to compute metrics, positional encoding and plot spatio-temporal graph.
• requirements.txt indicated the required libraries.

Usage

> python3 train_nn.py --help
usage: train_nn.py [-h] [--results_dir RESULTS_DIR] [--name NAME] [--gpu_ids GPU_IDS] [--checkpoint CHECKPOINT] [--config CONFIG] [--num_workers NUM_WORKERS] [--seed SEED]

options:
  -h, --help            show this help message and exit
  --results_dir RESULTS_DIR, -o RESULTS_DIR
                        models are saved here (default: results_directory)
  --name NAME           name of the experiment. It decides where to store samples and models (default: name_exp)
  --gpu_ids GPU_IDS     gpu ids: e.g. 0 0,1 (default: 0)
  --checkpoint CHECKPOINT
                        Path of a pretrained model to load weights before resuming training. If None random weights are applied (default: None)
  --config CONFIG       Path to the config file. (default: config/mlp.yaml)
  --num_workers NUM_WORKERS
  --seed SEED

The model weights, Tensorboard tracks and logs outputs to the results_directory after the training. Similar command can be used for Random Forest script.

Datasets

To build spatio-temporal graphs from scratch, download the Dynamic Earth Net dataset from here. Then copy the files to datasets/den/ or update the data path on the config files. Note that for the paper we only keep the first date of each month to stay in a fully supervised context.

To use the precomputed spatio-temporal graphs from our paper, download Graph Dynamic Earth Net data in Zenodo and move them to the datasets/den/processed/ folder. The data already provided in GitHub are sufficient to reproduce the results, but not to reconstruct the datacube and label the region. For that, you have to recompute the graphs by deleting the processed/ folder and run the script.

The spatio-temporal graphs are structured as follow, derived from the HeteroData class of PyG:

STG(
  img_dir='path/to/original/SITS/folder/',
  label_dir='path/to/the/corresponding/labels/folder/',
  num_nodes,      # Number of nodes in the spatio-temporal graph
  superpixels,    # The correspondance map to link nodes with the region position (Not provided in the precomputed graphs).
                  #   Use the ```reconstruct_to_datacube``` method to reconstruct the datacube from the graph when the superpixels map exists.
  EigVals,        # For node positional encoding
  EigVecs,        # For node positional encoding
  region={
    x,            # Node features
    y,            # Node labels
    pos,          # Node position in a Euclidean space
    node_index    # Node id
  },
  (region, spatial, region)={
    edge_index,   # Nodes from the same date linked by edges 
    edge_attr     # Spatial edge features
  },
  (region, temporal, region)={
    edge_index,   # Nodes from two consecutive dates linked by edges 
    edge_attr     # Temporal edge features
  }
)

Note that you can build spatio-temporal graphs and use this code with your own SITS dataset. Some public SITS datasets ready-to-download are listed here.

License Information

The data derived from Dynamic Earth Net data cubes are open sourced and ruled by the license CC BY-SA 4.0.

Reproducibility

The split used for the data is given in folder datasets/den/splits/. The seeds used for the 10 random initializations are 0,1,2,3,4,5,6,7,8,9.

Citation

If you use this work, consider citing our paper:

@inproceedings{dufourg2023graphdynamicearthnet,
  author={Dufourg, Corentin and Pelletier, Charlotte and May, Stéphane and Lefèvre, Sébastien},
  booktitle={IGARSS 2023 - 2023 IEEE International Geoscience and Remote Sensing Symposium}, 
  title={Graph Dynamic Earth Net: Spatio-Temporal Graph Benchmark for Satellite Image Time Series}, 
  year={2023},
  pages={7164-7167},
  doi={10.1109/IGARSS52108.2023.10281458}
}