Skip to content

Commit

Permalink
Merge branch 'main' of github.com:MinaFoundation/helm-charts into PM-…
Browse files Browse the repository at this point in the history 
…1603-redisinsight-helm-chart
  • Loading branch information
@Smorci
Smorci committed May 21, 2024
2 parents 77092fe + 7cb25f9 commit 2d3c8e2
Show file tree
Hide file tree
Showing 33 changed files with 1,238 additions and 625 deletions.
8 changes: 7 additions & 1 deletion .pre-commit-config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,13 @@ repos:
- id: helm-docs-built
args:
# Comma separated list, no space
- --chart-to-generate=gpt-survey-summarizer,submission-report,mina-transactions-generator,mina-payouts-data-provider,redisinsight
- --chart-to-generate=gpt-survey-summarizer,submission-report,mina-transactions-generator,mina-payouts-data-provider,mina-archive,redisinsight

# The `./` makes it relative to the chart-search-root
- --template-files=./README.md.gotmpl

- repo: https://github.com/shellcheck-py/shellcheck-py
rev: v0.10.0.1
hooks:
- id: shellcheck
files: mina-archive
6 changes: 3 additions & 3 deletions mina-archive/Chart.lock
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
dependencies:
- name: postgresql
repository: https://charts.bitnami.com/bitnami
version: 12.2.5
digest: sha256:4a76981e883b1ed7e0f2260dfc83f5873f7723033c056b698ece101c73d86d71
generated: "2023-03-20T15:58:24.567624941+02:00"
version: 15.2.9
digest: sha256:87fdb1b7bb010b959256f63b428b4729a98d3517c8af9feda36b7286ae399b7a
generated: "2024-05-09T09:25:38.19826-04:00"
4 changes: 2 additions & 2 deletions mina-archive/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ apiVersion: v2
name: mina-archive
description: A Helm chart for Mina Protocol's berkeley archive node
type: application
version: 2.0.0
version: 3.0.0
appVersion: 2.0.0
dependencies:
- name: postgresql
version: "*"
repository: "https://charts.bitnami.com/bitnami"
condition: archive.enablePostgresDB
condition: postgresql.enabled
keywords:
- archive
- postgres
Expand Down
182 changes: 86 additions & 96 deletions mina-archive/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,119 +1,109 @@
# `mina-archive` helm chart
# mina-archive

A helm chart to deploy Mina protocol archive node and Postgres database subchart.
![Version: 3.0.0](https://img.shields.io/badge/Version-3.0.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 2.0.0](https://img.shields.io/badge/AppVersion-2.0.0-informational?style=flat-square)

> **Note** Currently MF does not have chart repository. To install this chart i.e. with helmfile you need to reffer to it following ways:
A Helm chart for Mina Protocol's berkeley archive node

```console
# helmfile.yaml
<..>
releases:
- name: mina-archive
chart: git::ssh://[email protected]/MinaFoundation/helm-charts.git@mina-archive?ref=main
<..>
```

## Subcharts
**Homepage:** <https://minaprotocol.com/>

`mina-archive` uses only one subchart:
- `postgresql` from https://charts.bitnami.com/bitnami
## Requirements

> **Note** `mina-archive` is filled by the mina daemon connecting to it. However charts were created separately for wider usability. These dependencies can/should be glued outside of chart using tools like `helmfile`.
helm chart installed alongside found in https://github.com/minaProtocol/mina-helm-charts-private
| Repository | Name | Version |
|------------|------|---------|
| https://charts.bitnami.com/bitnami | postgresql | * |

## Prerequisites

Before installing this Helm chart, you should have the following prerequisites:
Before using this Helm chart, you should have the following prerequisites:

- Access to Kubernetes cluster
- Helm installed on your local machine
- Basic knowledge of Kubernetes and Helm
- Access to https://github.com/minaProtocol/mina-helm-charts-private
- Optional: helmfile to install this chart
- Access to Kubernetes cluster (If needed contact your friendly neighbourhood DevOps engineer)
- Helm >= v3.14.3
- (**Optional**) helmfile >= v0.162.0 to install this chart

## Installation

To install this Helm chart, the easiest is to create a helmfile.yaml with needed values and run:
```bash
$ helmfile template
$ helmfile apply
```
> Note: **examples** can be found in the repository
Or use helmfile only to generate resources and apply them with kubectl like so:
To install this Helm chart, the easiest is to create a helmfile.yaml with needed values and run:

```bash
$ helmfile template | kubectl -f -
```
helmfile template
helmfile apply
```

You can get some inspiration from helmfiles in `examples` folder.

Verify that the chart is deployed successfully:
Or use helmfile only to generate resources and apply them with kubectl like so:

```bash
helmfile status #although kubectl probably would give better insights.
```
helmfile template | kubectl -f -
```

## Configuration
Verify that the chart is deployed successfully:

To get all available values in cloned `mina-helm-charts-private` do:
> Note: `kubectl` is a better suited tool for this
```bash
helm show values ./mina-archive
```
helmfile status
```

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| databaseName | string | `"archive"` | Database name of the archive node |
| dbBootstrap.annotations | object | `{}` | Annotations to apply to the job |
| dbBootstrap.createDatabase | bool | `true` | Instanciate the database on the database server |
| dbBootstrap.enabled | bool | `false` | Enable to dbBootstrap job to populate the database schema or dump |
| dbBootstrap.extraSqlFileUrls | list | `[]` | SQL file urls to pre-download before executing the SQL file urls |
| dbBootstrap.maxExpectedDurationInSeconds | int | `1800` | Set the bootstrap duration expected to be used by other pods when waiting for the bootstrap to complete, before reaching timeout |
| dbBootstrap.podAnnotations | object | `{}` | Annotations to apply to the pod |
| dbBootstrap.postCustomSql | string | `"ALTER DATABASE {{ .Values.databaseName }} SET DEFAULT_TRANSACTION_ISOLATION TO SERIALIZABLE"` | Execute SQL inline command after loading the SQL file urls |
| dbBootstrap.sqlFileUrls | list | `[]` | SQL file urls to execute |
| dumpExporter.enabled | bool | `true` | Enabled dump exporter |
| dumpExporter.podAnnotations | object | `{}` | Annotations to the dump exporter |
| dumpExporter.s3.bucket | string | `""` | S3 bucket to export the dump to |
| dumpExporter.schedule | string | `"@midnight"` | Frequency to execute the dump exporter |
| dumpExporter.suspend | bool | `false` | Suspend the dump exporter execution |
| dumpExporter.ttlSecondsAfterFinished | int | `86400` | Seconds before cleaning up the dump exporter execution |
| externalDatabase.enabled | bool | `false` | |
| externalDatabase.host | string | `"host"` | Host for external database connection |
| externalDatabase.password | string | `"password"` | Password of external database connection |
| externalDatabase.port | int | `5432` | Port of external database connection |
| externalDatabase.username | string | `"username"` | Username of external database connection |
| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
| image.repository | string | `"gcr.io/o1labs-192920/mina-archive"` | Docker image repository |
| image.tag | string | `"2.0.0berkeley-rc1-1551e2f-focal"` | Docker image tag |
| missingBlocksGuardian.autoImportBlockUrl | string | `""` | URL to auto import a block when running the missing blocks guardian |
| missingBlocksGuardian.enabled | bool | `false` | Enabled missing blocks guardian |
| missingBlocksGuardian.podAnnotations | object | `{}` | Annotations to the missing blocks guardian |
| missingBlocksGuardian.precomputedBlocksUrl | string | `""` | URL to fetch the pre-computed blocks from |
| missingBlocksGuardian.schedule | string | `"@hourly"` | Frequency to execute the missing blocks guardian |
| missingBlocksGuardian.suspend | bool | `false` | Suspend the missing blocks guardian execution |
| missingBlocksGuardian.ttlSecondsAfterFinished | int | `86400` | Seconds before cleaning up the missing blocks guardian execution |
| network | string | `"network"` | Mina network name (e.g.: `mainnet`, `devnet`) |
| node.configFileUrl | string | `""` | Config file url to be downloaded and used as config file before the server starts |
| node.extraArgs | list | `[]` | Extra arguments for the mina archive process |
| node.extraEnvVars | list | `[]` | Extra environment variables for the mina archive process |
| node.metrics.enabled | bool | `true` | Enable metric service |
| node.podAnnotations | object | `{}` | Annotations to the mina archive pods |
| node.ports.metrics | int | `10002` | Mina archive metric port number |
| node.ports.rpc | int | `3086` | Mina archive RPC port number |
| node.readinessProbe | object | `{"exec":{"command":["bash","/scripts/archive-readiness.sh"]}}` | Readiness probe configuration |
| node.replicas | int | `1` | Replicas number for the archive node deployment |
| node.resources | object | `{}` | Resources for the mina archive pods |
| node.service.annotations | object | `{}` | Annotations to the mina archive service |
| node.service.labels | object | `{}` | Labels to the mina archive service |
| nodeSelector | object | `{}` | Node selector for all the pods |
| postgresClientDockerImage | string | `"bitnami/postgresql:16.2.0-debian-12-r18"` | Image to use as postgresql client, to export dumps for example |
| postgresql.auth.enablePostgresUser | bool | `false` | Enable the default postgres user |
| postgresql.auth.password | string | `"password"` | Password for the database |
| postgresql.auth.username | string | `"username"` | Username for the database |
| postgresql.enabled | bool | `true` | Enable local postgresql database server |
| postgresql.primary.extendedConfiguration | string | `"max_wal_size=2048\n"` | Extended configuration to configure postgresql server |
| postgresql.primary.persistence.enabled | bool | `true` | Enable the persistence for the postgresql server |
| postgresql.primary.persistence.size | string | `"8Gi"` | Size of the postgresql server volume |
| postgresql.primary.persistence.storageClass | string | `""` | Storage class for the postgresql server volume |
| postgresql.primary.resourcesPreset | string | `"nano"` | Resources preset to set resource requests and limits |
| serviceAccount.annotations | object | `{}` | Service account annotations |
| tolerations | list | `[]` | Toleration for all the pods |

The following table lists the configurable parameters of the `mina-archive` chart and its common default values.

### Required Settings

Parameter | Description
--- | ---
`archive.testnet` | Which `testnet` archive will be running on.
`archive.image` | container image to use for operating an archive node
`archive.remoteSchemaAuxFiles` | list of schema file download urls needed to bootstrap database.
`archive.service.labels` | Custom Labels for the Archive Service | `{}`
`archive.service.annotations` | Custom Annotations for the Archive Service | `{}`

### Optional Settings

Parameter | Description | Default
--- | --- | ---
`archive.postgresHost` | Postgres database host to store archival data | `see [default] values.yaml`
`archive.postgresUri` | Postgres [connection URI](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) to access postgres datastore instance | `see [default] values.yaml`
`archive.bootstrapDB` | Run Bootstrap Job against the Postgres Database | `true`
`dumpCronjob.enabled` | Backup the database periodically | `false`
`dumpCronjob.s3Bucket` | s3 Bucket to upload backup | `673156464838-mina-archive-node-backups`
`archive.initFromDump.enabled` | Populate Postgres Database with an existing Dump | `false`
`archive.initFromDump.archiveDumpURL` | URL to retrieve SQL Dump | `https://storage.googleapis.com/mina-archive-dumps`
`archive.initFromDump.precomputedBlocksURL` | URL to retrieve Precomputed Blocks | `https://storage.googleapis.com/mina_network_block_data`
`archive.initFromDump.missingBlocksGuardianWaitTime` | Number of seconds before the Missing Blocks Guardian kicks off | `1200`
`archive.remoteSchemaFile` | archive database schema during initialization | `see [default] values.yaml`
`archive.metrics.enabled` | Whether to enable prometheus exporter for mina-archive | `false`
`archive.metrics.port` | Prometheus exporter port for mina-archive | `10002`
`serviceAccount.annotations` | Allow role to assume this service account | `{}`
`podAnnotations` | Custom Annotations for the pod | `{}`

### Resource allocation

Parameter | Description | Default
--- | --- | ---
`resources.memoryRequest` | RAM to claim for mina archive container | "6.0Gi"
`resources.cpuRequest` | # of CPUs to claim for mina archive container | "3"
`resources.memoryLimit` | RAM limit for mina archive container | "8.0Gi"
`resources.cpuLimit` | # of CPUs limit for mina archive container | "4"

### Health probes

`mina-archive` health is determined checking it's rpc port liveness.

Parameter | Description | Default
--- | --- | ---
`healthcheck.startup.periodSeconds` | How often startupProbe is checked | `30`
`healthcheck.startup.failureThreshold` | # times startupProbe is allowed to fail | `5`
`healthcheck.failureThreshold` | # times liveness/readiness is allowed to fail | `5`
`healthcheck.periodSeconds` | How often liveness/readiness probes are checked | `5`
`healthcheck.initialDelaySeconds` | Time to wait before start checking liveness/readiness status | 30

### Postgresql Configuration

Please refer to official Postgresql by Bitnami [Documentation](https://github.com/bitnami/charts/blob/main/bitnami/postgresql/README.md#parameters)
Binary file removed mina-archive/charts/postgresql-12.2.5.tgz
View file
Binary file not shown.
Binary file added mina-archive/charts/postgresql-15.2.9.tgz
View file
Binary file not shown.
48 changes: 48 additions & 0 deletions mina-archive/examples/dev.values.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
network: qanet
databaseName: archive
image:
repository: gcr.io/o1labs-192920/mina-archive
tag: 2.0.0berkeley-rc1-1551e2f-focal

serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::673156464838:role/MinaNodesS3_sandbox

postgresql:
enabled: true
primary:
resourcesPreset: "2xlarge"
persistence:
enabled: false
storageClass: "ebs-gp3-encrypted"
size: 8Gi

dbBootstrap:
enabled: true

createDatabase: true
sqlFileUrls:
- https://raw.githubusercontent.com/MinaProtocol/mina/develop/src/app/archive/create_schema.sql
extraSqlFileUrls:
- https://raw.githubusercontent.com/MinaProtocol/mina/develop/src/app/archive/zkapp_tables.sql

# createDatabase: false
# sqlFileUrls:
# - https://673156464838-mina-archive-node-backups.s3-us-west-2.amazonaws.com/qanet/qanet-archive-dump-[DATE]_0000.sql.tar.gz

node:
replicas: 1
configFileUrl: http://673156464838-mina-genesis-ledgers.s3-us-west-2.amazonaws.com/qanet/genesis_ledger.json

missingBlocksGuardian:
enabled: true
suspend: true
precomputedBlocksUrl: https://673156464838-mina-precomputed-blocks.s3-us-west-2.amazonaws.com/qanet
autoImportBlockUrl: https://673156464838-mina-precomputed-blocks.s3-us-west-2.amazonaws.com/qanet/qanet-500-3NKA1RcRkYrPmASYvhRuW68N8a9p6kRjpCaWxUj67nKwShZ76ZHB.json
# autoImportBlockUrl: https://673156464838-mina-precomputed-blocks.s3.us-west-2.amazonaws.com/qanet/qanet-34205-3NKnu7LiJ627UWpT3g7y4xFQSCAQnYHQehyRQ51eibam53XgpQFh.json

dumpExporter:
enabled: false
suspend: true
s3:
bucket: 673156464838-mina-archive-node-backups
45 changes: 0 additions & 45 deletions mina-archive/examples/helmfile.yaml
View file

This file was deleted.

36 changes: 36 additions & 0 deletions mina-archive/examples/mainnet.values.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
network: mainnet
databaseName: archive_balances_migrated
image:
repository: minaprotocol/mina-archive
tag: 1.4.1-e76fc1c-bullseye

serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::673156464838:role/MinaNodesS3_sandbox

postgresql:
enabled: true
primary:
resourcesPreset: "2xlarge"
persistence:
enabled: true
storageClass: "ebs-gp3-encrypted"
size: 48Gi

dbBootstrap:
enabled: true

createDatabase: false
sqlFileUrls:
# - https://673156464838-mina-archive-node-backups.s3-us-west-2.amazonaws.com/mainnet/mainnet-archive-dump-[DATE]_0000.sql.tar.gz
- https://673156464838-mina-archive-node-backups.s3.us-west-2.amazonaws.com/mainnet/mainnet-archive-dump-2024-04-28_0000.sql.tar.gz
# - https://storage.googleapis.com/mina-archive-dumps/backup-mainnet-pre-hf-dry-run-2-migrated-dump-2024-05-02_0700.sql.tar.gz

node:
replicas: 1

missingBlocksGuardian:
enabled: true
suspend: true
precomputedBlocksUrl: https://673156464838-mina-precomputed-blocks.s3.us-west-2.amazonaws.com/mainnet
autoImportBlockUrl: https://673156464838-mina-precomputed-blocks.s3.us-west-2.amazonaws.com/mainnet/mainnet-350000-3NLGassgPE49SrxMtvz5buEukiAZJjNHSJaqo3cmUjbUfDC8Jrm3.json
14 changes: 14 additions & 0 deletions mina-archive/scripts/archive-command-init.sh
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!/usr/bin/env bash

set -e

echo "INFO: Archive init script started"

/scripts/wait-for-db.sh

if [[ $ARCHIVE_CONFIG_FILE_URL != "" ]]; then
echo "INFO: Downloading config file $ARCHIVE_CONFIG_FILE_URL"
curl -s "$ARCHIVE_CONFIG_FILE_URL" > /config/config-file.json
fi

echo "INFO: Archive init script complete"
Loading

0 comments on commit 2d3c8e2

Please sign in to comment.