SB5.1 - OCI - Inconsistency between Core/CassandraBackup and DP/CassandraBackup

[heungheung]

I am not 100% sure whether is should be installation, configuration or a general question.... will try to ask here first

Core/CassandraBackup is using this Jenkins job
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/deploy/jenkins/jobs/OpsAdministration/jobs/dev/jobs/Core/jobs/CassandraBackup/config.xml
with repo
https://github.com/project-sunbird/sunbird-devops.git
Jenkins file
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/pipelines/backup/cassandra-backup/Jenkinsfile
which will use this playbook
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/ansible/cassandra-backup.yml
eventually invoke role
https://github.com/project-sunbird/sunbird-devops/tree/release-5.1.0/ansible/roles/cassandra-backup
with addn args --disablesnapshot, and
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/ansible/roles/cassandra-backup/tasks/main.yml#L12
line 12 will use this py
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/deploy/cassandra_backup.py#L62
line 62, tgz disabled

DP/CassandraBackup is using this Jenkins job
https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/deploy/jenkins/jobs/OpsAdministration/jobs/dev/jobs/DataPipeline/jobs/CassandraBackup/config.xml
with repo
https://github.com/project-sunbird/sunbird-data-pipeline.git
Jenkins file (or other branch)
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-5.2.0/pipelines/backup/cassandra-backup/Jenkinsfile
with the playbook
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-5.2.0/ansible/cassandra-backup.yml
role
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/tree/release-5.2.0/ansible/roles/cassandra-backup
and which DOES us the j2 template
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-5.2.0/ansible/roles/cassandra-backup/tasks/main.yml#L5
with
https://github.com/Sunbird-Obsrv/sunbird-data-pipeline/blob/release-5.2.0/ansible/roles/cassandra-backup/templates/cassandra_backup.j2#L23
and Line 23, there is tgz.

Here are the questions, why there are differences in terms of

1. using tar giz or not
2. using snapshot or not

[beepdot]

Hi @heungheung

In sunbird we run two separate cassandra clusters. One for Core module and the other for Data Pipeline module. In the installation docs we tell to use only one cassandra cluster (single node) to save cost. But that is not recommended for production use.

In our case the Core cassandra data (600GB+ per node) and server size(7-14 nodes) is quite big. So we take a snapshot and directly upload the snapshot folder to storage account. This saves us time and additional disk required to gz it. Azcopy tool is quite fast to upload the entire directory to storage.

During restore also its quite easy to restore this data back just by placing the folders into respective directories and running a few commands. This restore approach works only if the number of nodes are same (backup cluster nodes = restore cluster nodes) as we use the token ring approach to match the data ownership on each node.

The restore steps are mentioned here - project-sunbird/sunbird-devops#2443

The DP cassandra is a single node cluster and also has a small data footprint. So we take snapshot and store it as a gz file. We can use the Core cassandra backup approach to the DP cassandra also if data or nodes grow in size. But we haven't felt the need for it till date.

Hope this answers your query.

[gandham-santhosh]

Adding to what @keshavprasadms mentioned

• Zipping the files step was made optional as process of zipping takes lot of time and consumes resources on the running cassandra cluster which lead to lower response times and data inconsistencies at peak traffic times. Also unnecessarily consumes the disk on the running node
• token ring approach was taken instead nodetool snapshot as the restoring the whole cluster to a new cluster process is easy and quick
  However, adopter can use the approach that works for their usecase or need.

[heungheung]

yes and understand... just wondering.... the original model is j2 and template the backup script, then run it. Would it be nice if how the j2 got templated is control by the ansible inventory file? Instead of using the fixed .py? https://github.com/project-sunbird/sunbird-devops/blob/release-5.1.0/deploy/cassandra_backup.py#L62

Totally agree adopter can choose & use the approach that fits for their use case, was just think how to help adopter easily get the approach deployed. Thanks!

[beepdot]

As the python script does not include any ansible variable blocks, a jinja template is not required. But if it has inventory dependencies, then yes we can use the j2 template.