Charts[Network Node]: Ensure Mirror Uploader Uploads Stream Files To S3 Compatible Minio Bucket

[leninmehedy]

User story

As a FST user
I want
various node stream files be uploaded into a S3 compatible bucket (compressed/uncompressed)
So that
I can run mirror node using those files later

Assumptions

• User is running a local kind cluster
• User is able to run helm and kubectl command
• User is able to deploy minio using helm chart

Requirements

• Setup buckets using Minio
• Configure mirror uploader to use minio buckets
• Ensure it is uploading files (compressed/uncompressed)
• Optional: Ensure those stream files are not corrupted (can be used to start mirror nodes)

Notes

• Run network node (use the bash scripts that we are going to create to install and run node software)
• Ensure it is generating stream files
• Ensure stream files are being uploaded to the bucket (compressed/uncompressed)
• Optional: start the mirror node container using these stream files (See the testing section in the comment below)

[nathanklick]

Further Context

MinIO splits deployments into two Helm charts, the operator and the tenant. The operator chart must only be deployed once per K8S cluster. The tenant chart may be deployed once per namespace.

Our intent for utilization

• Deploy the operator chart once per cluster:
  • For local internal engineer testing, we should update the Makefile definitions to provide a target for deploying the operator. Within the scope of this ticket.
  • For our persistent GKE cluster for nightly tests, we should deploy and manage the operator chart via ArgoCD during cluster spin up. OUTSIDE the scope of this ticket.
  • For our normal users, the Java side of FST should detect the presence or absence of the operator chart/deployment and deploy the operator chart if absent from the targeted K8S cluster. OUTSIDE the scope of this ticket.
• Deploy the tenant chart once per Hedera network deployment:
  • For all cases, we should add the tenant chart as a conditional subchart to our hedera-network chart and provide reasonable default value overrides. Please see this Helm chart guide and also this StackOverflow article for further details about conditional subcharts. Within the scope of this ticket.

Testing Requirements

• Mirror Node Tests
  • Manually provision a Mirror Node deployment using the production chart as documented here.
    • Ensure the Mirror Node is able to consume the files correctly from the MinIO server.
      • This will validate our mirror.py configuration and ensure all bucket paths are correct with regards to naming and structure.

User Mentions

• @leninmehedy
• @deepak-swirlds

[swirlds-automation]

🎉 This issue has been resolved in version 0.7.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀