Export files to s3 and crons to scripts #368
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WARNING - pre-requisite : before merging this PR, need to have deployed the build-addok-bundle cloud service.
Context
In order to make ban-plateforme cloud ready, we need to modify some mechanisms that won't be possible on the cloud. This PR is targeting the file storage system. For now, the storage used in the server storage system. We need to migrate to an S3 storage system. As a consequence, we need to migrate the way export files are created and stored. The export files targeted are :
1- Department files (csv, csv-bal, csv-with-ids, addok)
2- National files :
Enhancements
1- Departement files : modify the current behavior that write the files on the server file storage to a behavior that does not right anything on the server but that upload the files on a S3. In order to work locally in dev mode, this PR also adds a "minio" container to the docker-compose.
2- National files : modify the current behavior using crons outside ban-plateforme (on the server) to a behavior that starts scripts using crons on the ban-plateforme worker.
How to test
1- Clone this branch.
2- Enter the right values for the following env variables :
3- Start the docker compose :
docker compose up --build -d
4- For department export files, start :
yarn dist
5- For national files, start :
node scripts/create-and-upload-national-files-to-s3.js