Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export files to s3 and crons to scripts #368

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

antoineludeau
Copy link
Member

@antoineludeau antoineludeau commented Feb 13, 2024

WARNING - pre-requisite : before merging this PR, need to have deployed the build-addok-bundle cloud service.

Context

In order to make ban-plateforme cloud ready, we need to modify some mechanisms that won't be possible on the cloud. This PR is targeting the file storage system. For now, the storage used in the server storage system. We need to migrate to an S3 storage system. As a consequence, we need to migrate the way export files are created and stored. The export files targeted are :
1- Department files (csv, csv-bal, csv-with-ids, addok)
2- National files :

  • csv : adresses-france.csv.gz, lieux-dits-beta-france.csv.gz
  • csv-with-ids : adresses-with-ids-france.csv.gz, lieux-dits-with-ids-beta-france.csv.gz
  • csv-bal : adresses-france.csv.gz
  • addok : adresses-addok-france.ndjson.gz, addok-france-bundle.zip

Enhancements

1- Departement files : modify the current behavior that write the files on the server file storage to a behavior that does not right anything on the server but that upload the files on a S3. In order to work locally in dev mode, this PR also adds a "minio" container to the docker-compose.

2- National files : modify the current behavior using crons outside ban-plateforme (on the server) to a behavior that starts scripts using crons on the ban-plateforme worker.

How to test

1- Clone this branch.
2- Enter the right values for the following env variables :

MINIO_ROOT_USER=minio
MINIO_ROOT_PASSWORD=minio123
MINIO_PORT=9000
MINIO_FRONT_PORT=9001

3- Start the docker compose :
docker compose up --build -d

4- For department export files, start :
yarn dist

5- For national files, start :
node scripts/create-and-upload-national-files-to-s3.js

@antoineludeau antoineludeau force-pushed the antoineludeau/export-files-to-S3 branch 2 times, most recently from 1cd5112 to dafbe96 Compare March 4, 2024 14:45
@antoineludeau antoineludeau force-pushed the antoineludeau/export-files-to-S3 branch from dafbe96 to 2b84afa Compare March 5, 2024 15:59
@antoineludeau antoineludeau marked this pull request as ready for review March 6, 2024 16:00
@nkokla nkokla marked this pull request as draft April 4, 2024 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant