Data transfer to MCP

[slesaad]

Epic

#118

Description

One of the most important tasks in migration of VEDA to MCP is migration of the data files to the MCP s3 bucket.

Based on decisions made in #118, it's been decided that this will be carried out by running the data transformation and ingestion pipeline with a flag set to transfer the data to the MCP s3 bucket.

A role based policy will be used to gain access to the MCP s3 bucket from the ingestion pipeline.

Examples

The current data link points to a UAH bucket. That's where the data files exist.

{
  "assets": {
    "cog_default": {
      "href": "s3://climatedashboard-data/bmhd_30m_monthly_bkp/finalBMHD_ScaledVenice_202203.tif",
      "type": "image/tiff; application=geotiff; profile=cloud-optimized",
      "roles": [
        "data",
        "layer"
      ]
    },
  }
}

At the end of the data migration, the links should look like the following and the data should exist in that link:

{
  "assets": {
    "cog_default": {
      "href": "s3://veda-data-store-staging/nightlights-hd-monthly/finalBMHD_ScaledVenice_202203.tif",
      "type": "image/tiff; application=geotiff; profile=cloud-optimized",
      "roles": [
        "data",
        "layer"
      ]
    },
  }
}

Acceptance Criteria:

• All the data is transferred to the MCP bucket
• All the data link points to the MCP s3 url
• The asset/data links from the staging catalog can still be viewed in the cog viewer after the transfer

Checklist for collections

• nightlights-hd-1band
• social-vulnerability-index-housing-nopop
• grdi-v1-built
• nightlights-hd-monthly
• MO_NPP_npp_vgpm
• HLSS30.002 (these are in lpdaac protected buckets; @anayeaye is working on creating our own version of these)
• HLSL30.002 ( same )
• OMI_trno2-COG
• social-vulnerability-index-household
• grdi-v1-raster
• OMSO2PCA-COG
• grdi-shdi-raster
• facebook_population_density
• grdi-vnl-slope-raster
• social-vulnerability-index-socioeconomic
• social-vulnerability-index-socioeconomic-nopop
• grdi-filled-missing-values-count
• grdi-vnl-raster
• grdi-cdr-raster
• blue-tarp-planetscope
• no2-monthly-diff
• social-vulnerability-index-household-nopop
• social-vulnerability-index-minority
• social-vulnerability-index-overall-nopop
• no2-monthly
• social-vulnerability-index-overall
• social-vulnerability-index-housing
• blue-tarp-detection
• IS2SITMOGR4-cog
• social-vulnerability-index-minority-nopop
• nceo_africa_2017
• geoglam
• grdi-imr-raster
• nightlights-hd-3bands

Checklist:

• Epic Link
• Detailed description
• Concept diagrams
• Assignee

[xhagrg]

We should probably push to prod bucket rather than staging bucket.

[slesaad]

Misc TODOs

• Remove povmap-grdi-v1_VNL-2020-01-01_2020-12-31 from grdi-vnl-raster
• Delete collection IS2SITMOGR4, added IS2SITMOGR4-cog instead
• Delete nightlights-hd-3bands
• Add nightlights-500m-daily

[slesaad]

Some pgstac database quirks ☠️ realised while migrating the datasets that we should be aware of:

• When you update a collection after the items have been ingested into the collection, the items disappear. 🛑 DONOT UPDATE A COLLECTION AFTER THE ITEMS ARE INGESTED, else you'll have to run the ingestion again.
• We can't use same id 👯 for items even if they are in different collection

[abarciauskas-bgse]

The data products are in s3://veda-data-store-staging and not s3://veda-data-store - @xhagrg @slesaad can we migrate the products to be in s3://veda-data-store? 🙏🏽

[abarciauskas-bgse]

We are having a longer discussion on slack so will follow up with next steps