GHGC STAC Ingestor

Important

The US GHG Center has started using veda-backend repository directly for its backend services, which includes the STAC ingestion services. Hence, this forked version of the veda-stac-ingestor repository is no longer maintained and so the repository is now archived.

This service provides an entry-point for users/services to add new records to our STAC database. Its primary functions are to 1) validate the STAC Items before insertion, 2) batch the insertions to reduce load on our STAC database.

STAC items are validated to ensure that:

1. It meets the STAC Specification
2. All assets are accessible
3. Its collection exists

Development

Configuration

This codebase utilizes the Pydantic SSM Settings module to attempt to load configuration from AWS SSM Parameter Store.

Running API

1. Create virtual environment:

   python3 -m venv .venv
   source .venv/bin/activate

2. Install dependencies:

   pip install -r api/requirements.txt
   

3. Run API:

   python api/local.py
   

   Note: If no .env file is present, the API will connect to resources in the dev deployment via pydantic-ssm-settings. This requires that your AWS_PROFILE be set to the profile associated with the AWS account hosting the dev deployment.

Using the API

Please go through the API Usage docs for a guide on ingesting and publishing data to the GHGC data store & STAC API.

Deployment

Fetch environment variables using AWS CLI

To retrieve the variables for a stage that has been previously deployed, the secrets manager can be used to quickly populate an .env file.

Note: The environment variables stored as AWS secrets are manually maintained and should be reviewed before using.

export AWS_SECRET_ID=stac-ingestor-env-secret-<stage>

aws secretsmanager get-secret-value --secret-id ${AWS_SECRET_ID} --query SecretString --output text | jq -r 'to_entries|map("\(.key)=\(.value|tostring)")|.[]' > .env

This script is also available at scripts/sync_env.sh, which can be invoked with the secret name as an argument:

. scripts/sync_env.sh stac-ingestor-env-secret-<stage>

License

This project is licensed under Apache 2, see the LICENSE file for more details.