Skip to content

Commit

Permalink
Merge pull request #8646 from nimrod-becker/backport_to_5_18
Browse files Browse the repository at this point in the history
Backport to 5.18 : CI and Online Upgrade
  • Loading branch information
nimrod-becker authored Jan 7, 2025
2 parents af4290e + e14b749 commit 55d3c09
Show file tree
Hide file tree
Showing 14 changed files with 499 additions and 124 deletions.
79 changes: 79 additions & 0 deletions .github/workflows/Validate-package-lock.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Validate package-lock.json Tests
on: [push, pull_request]

jobs:
run-package-lock-validation:
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up Node.js
uses: actions/setup-node@v3
with:
node-version: 22

- name: Backup the current package-lock.json
run: |
# Backup the current package-lock.json
mv package-lock.json package-lock-backup.json
# Generate a new package-lock.json
npm install
- name: Validate top-level versions in package-lock.json
run: |
# Validate the main version field
top_version_backup=$(jq -r '.version' package-lock-backup.json)
top_version_new=$(jq -r '.version' package-lock.json)
# Define the ANSI escape code for red
RED='\033[0;31m'
NC='\033[0m' # No Color (resets the color)

if [ "$top_version_backup" != "$top_version_new" ]; then
echo "The top-level version in package-lock.json is inconsistent."
echo -e "${RED}Original version: $top_version_backup${NC}"
echo -e "${RED}Generated version: $top_version_new${NC}"
exit 1
fi

- name: Validate dependencies top-level versions in package-lock.json
run: |
# Extract and validate top-level module versions
jq '.packages[""].dependencies' package-lock-backup.json > top-level-versions-backup.json
jq '.packages[""].dependencies' package-lock.json > top-level-versions-new.json
if ! diff -q top-level-versions-backup.json top-level-versions-new.json > /dev/null; then
echo -e "${RED}Top-level module versions in package-lock.json are inconsistent.${NC}"
echo -e "${RED}Differences:${NC}"
diff top-level-versions-backup.json top-level-versions-new.json || true
exit 1
else
echo "Top-level module versions are consistent. Validation passed."
fi

- name: Validate devDependencies top-level versions in package-lock.json
run: |
# Extract and validate top-level module versions
jq '.packages[""].devDependencies' package-lock-backup.json > top-level-versions-backup.json
jq '.packages[""].devDependencies' package-lock.json > top-level-versions-new.json

# Define the ANSI escape code for red
RED='\033[0;31m'
NC='\033[0m' # No Color (resets the color)

if ! diff -q top-level-versions-backup.json top-level-versions-new.json > /dev/null; then
echo -e "${RED}Top-level module versions in package-lock.json are inconsistent.${NC}"
echo -e "${RED}Differences:${NC}"
diff top-level-versions-backup.json top-level-versions-new.json || true
exit 1
else
echo "Top-level module versions are consistent. Validation passed."
fi

168 changes: 163 additions & 5 deletions docs/NooBaaNonContainerized/Upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
1. [Introduction](#introduction)
2. [General Information](#general-information)
3. [Download Upstream RPM](#download-upstream-rpm)
4. [Offline Upgrade](#offline-upgrade)
4. [Offline Upgrade (Version < 5.18.0)](#offline-upgrade-version--5180)
1. [Offline Upgrade steps](#offline-upgrade-steps)
5. [Online Upgrade](#online-upgrade)
5. [Online Upgrade (Version >= 5.18.0)](#online-upgrade-version--5180)


## Introduction
Expand All @@ -29,7 +29,7 @@ This document provides step-by-step instructions to help you successfully upgrad

For NooBaa upstream (open source code) RPM download instructions, See [NooBaa Non Containerized Getting Started](./GettingStarted.md).

## Offline Upgrade
## Offline Upgrade (Version < 5.18.0)
The currently available upgrade process of NooBaa Non Containerized is an offline upgrade. Offline upgrade means that NooBaa service must be stopped during the upgrade and that NooBaa endpoints won't be handling S3 requests at the time of the upgrade.

### Offline Upgrade steps
Expand Down Expand Up @@ -69,5 +69,163 @@ The currently available upgrade process of NooBaa Non Containerized is an offlin
cat /etc/noobaa.conf.d/system.json
{"hostname":{"current_version":"5.17.0","upgrade_history":{"successful_upgrades":[{"timestamp":1719299738760,"completed_scripts":[],"from_version":"5.15.4","to_version":"5.17.0"}]}}}
```
## Online Upgrade
The process of Online Upgrade of Non Containerized NooBaa is not supported yet.
## Online Upgrade (Version >= 5.18.0)

### Online Upgrade Goals
**1. Minimal downtime -** Ensure minimal downtime for each node.

**2. Incremental changes -** Split upgrade to small chunks, for example, upgrade nodes one by one. Each node will get upgraded on its turn, the other nodes will still be available for handling s3 requests.

**3. Rollback capability -** Mechanism for revert to the previous version in case something went wrong during the upgrade.

**4. Schema backward compatibility -** Changes to account/bucket/config schema must be backwards compatible to allow seamless transitions to new version.


### Online Upgrade Algorithm

1. Initiate config directory backup (#1).
2. Iterate nodes one by one -
* Stop NooBaa service (or suspend the node in CES)
* RPM upgrade each node.
* Restart NooBaa service on each node.
3. Wait for all hosts to finish RPM upgrade (source code upgrade).
4. Initiate config directory backup (#2).
5. Initiate upgrade of the config directory using a noobaa-cli complete upgrade command. (point of no return)

Online Upgrade Algorithm commands examples -
1. Config directory backup -
1. CES - `mms3 config backup /path/to/backup/location`
2. Non CES - `cp -R /etc/noobaa.conf.d/ /path/to/backup/location`
2. Stop NooBaa service - `systemctl stop noobaa`
3. RPM upgrade on a specific node - `rpm -Uvh /path/to/new_noobaa_rpm_version.rpm`
4. Restart NooBaa service - `systemctl restart noobaa`
5. `noobaa-cli upgrade start --expected_version=5.18.0 --expected_hosts=hostname1,hostname2,hostname3`

### Additional Upgrade Properties of `system.json`-

1. New per host property -
- config_dir_version

2. New config directory information -
- config_directory
- config_dir_version
- phase
- upgrade_package_version
- in_progress_upgrade - (during the upgrade)
- timestamp
- completed_scripts
- running_host
- config_dir_from_version
- config_dir_to_version
- package_from_version
- package_to_version
- upgrade_history
- last_failure (if last upgrade failed)
- successful_upgrades

#### system.json new information examples -
1. During Upgrade - `cat /etc/noobaa.conf.d/system.json | jq .`
```json
{
"my_host1":{
"current_version":"5.18.0",
"config_dir_version": "1.0.0",
"upgrade_history":{
"successful_upgrades":[{
"timestamp":1730890665481,
"from_version":"5.17.1",
"to_version":"5.18.0"
}]
}
},
"config_directory":{
"phase":"CONFIG_DIR_LOCKED", // <- config dir is locked during an upgrade
"config_dir_version":"0.0.0", // <- config_dir_version is still the old config_dir_version
"upgrade_package_version":"5.17.1", // <- upgrade_package_version is still the old upgrade_package_version
"in_progress_upgrade":[{ // <- in_progress_upgrade property during the upgrade
"timestamp":1730890691016,
"completed_scripts": [],
"running_host":"my_host1",
"config_dir_from_version":"0.0.0",
"config_dir_to_version":"1.0.0",
"package_from_version":"5.17.1",
"package_to_version":"5.18.0"
}]
}
}
```

2. After a successful upgrade - `cat /etc/noobaa.conf.d/system.json | jq .`
```json
{
"my_host1":{
"current_version":"5.18.0",
"config_dir_version": "1.0.0",
"upgrade_history":{
"successful_upgrades":[{
"timestamp":1730890665481,
"from_version":"5.17.1",
"to_version":"5.18.0"
}]
}
},
"config_directory":{
"phase":"CONFIG_DIR_UNLOCKED", // <- after a successful upgrade, config dir is unlocked
"config_dir_version":"1.0.0", // <- config_dir_version is the new config_dir_version
"upgrade_package_version":"5.18.0", // <- upgrade_package_version is the new upgrade_package_version
"upgrade_history":{ // <- a new item in the successful upgrades array was added
"successful_upgrades":[{
"timestamp":1730890691016,
"completed_scripts":
["/usr/local/noobaa-core/src/upgrade/nc_upgrade_scripts/1.0.0/config_dir_restructure.js"],
"running_host":"my_host1",
"config_dir_from_version":"0.0.0",
"config_dir_to_version":"1.0.0",
"package_from_version":"5.17.1",
"package_to_version":"5.18.0"
}]
}
}
}
```

3. After a failing upgrade - `cat /etc/noobaa.conf.d/system.json | jq .`
```json
{
"my_host1":{
"current_version":"5.18.0",
"config_dir_version": "1.0.0",
"upgrade_history":{
"successful_upgrades":[{
"timestamp":1730890665481,
"from_version":"5.17.1",
"to_version":"5.18.0"
}]
}
},
"config_directory":{
"phase":"CONFIG_DIR_LOCKED", // <- after a failing upgrade, config dir is still locked
"config_dir_version":"0.0.0", // <- config_dir_version is still the old config_dir_version
"upgrade_package_version":"5.17.1", // <- upgrade_package_version is still the old upgrade_package_version
"upgrade_history":{
"successful_upgrades": [], // <- successful_upgrades array is empty/doesn't contain the failed upgrade
"last_failure":{ // <- a last_failure property is set in upgrade history
"timestamp":1730890676741,
"completed_scripts":[
"/usr/local/noobaa-core/src/upgrade/nc_upgrade_scripts/1.0.0/config_dir_restructure.js"],
"running_host":"my_host1",
"config_dir_from_version":"0.0.0",
"config_dir_to_version":"1.0.0",
"package_from_version":"5.17.1",
"package_to_version":"5.18.0",
"error": "Error: _run_nc_upgrade_scripts: nc upgrade manager failed!!!, Error: this is a mock error\n at NCUpgradeManager._run_nc_upgrade_scripts (/usr/local/noobaa-core/src/upgrade/nc_upgrade_manager.js:258:19)\n at async NCUpgradeManager.upgrade_config_dir (/usr/local/noobaa-core/src/upgrade/nc_upgrade_manager.js:119:13)\n at async start_config_dir_upgrade (/usr/local/noobaa-core/src/manage_nsfs/upgrade.js:52:29)\n at async Object.manage_upgrade_operations (/usr/local/noobaa-core/src/manage_nsfs/upgrade.js:22:13)\n at async main (/usr/local/noobaa-core/src/cmd/manage_nsfs.js:73:13)"
}
}
}
}
```
### Upgrade Helpers
1. NooBaa Health CLI - will report on the config directory status, upgrade failures and hosts that are blocked for config directory updates.
2. NooBaa CLI upgrade status - will print the upgrade status per the information written in system.json.
2 changes: 1 addition & 1 deletion src/cmd/nsfs.js
Original file line number Diff line number Diff line change
Expand Up @@ -330,7 +330,7 @@ async function main(argv = minimist(process.argv.slice(2))) {
const nc_upgrade_manager = new NCUpgradeManager(config_fs);
await nc_upgrade_manager.update_rpm_upgrade();
} else {
system_data = await config_fs.init_nc_system();
system_data = await config_fs.register_hostname_in_system_json();
}
}

Expand Down
47 changes: 45 additions & 2 deletions src/manage_nsfs/health.js
Original file line number Diff line number Diff line change
Expand Up @@ -450,15 +450,17 @@ class NSFSHealth {
*/
_get_config_dir_status(system_data) {
if (!system_data) return { error: 'system data is missing' };
const blocked_hosts_status = this._get_blocked_hosts_status(this.config_fs, system_data);
const config_dir_data = system_data.config_directory;
if (!config_dir_data) return { error: 'config directory data is missing, must upgrade config directory' };
if (!config_dir_data) return { error: 'config directory data is missing, must upgrade config directory', blocked_hosts: blocked_hosts_status };
const config_dir_upgrade_status = this._get_config_dir_upgrade_status(config_dir_data);
return {
phase: config_dir_data.phase,
config_dir_version: config_dir_data.config_dir_version,
upgrade_package_version: config_dir_data.upgrade_package_version,
upgrade_status: config_dir_upgrade_status,
error: config_dir_upgrade_status.error || undefined
error: config_dir_upgrade_status.error || undefined,
blocked_hosts: blocked_hosts_status
};
}

Expand All @@ -480,6 +482,47 @@ class NSFSHealth {
}
}

/**
* _get_blocked_hosts_status checks if there are any hosts blocked for updating the config directory
* 1. if only config dir was upgraded (>=5.18.0) - host's config_dir_version does not exist (<5.18.0) and system's config_dir_version exists (>=5.18.0)
* it means it's not blocked because the source code won't include _throw_if_config_dir_locked() but it still can create invalid config files, therefore including it in the blocked list
* 2. if system's config_dir_version wasn't upgraded yet and hosts's config_dir_version exist (>= 5.18.0)
* it means updates to the config directory from this host are blocked
* 3. if system's config dir version does not match the hosts's config_dir_version - updates to the config directory from this host are blocked
* @param {import('../sdk/config_fs').ConfigFS} config_fs
* @param {Object} system_data
* @returns {Object}
*/
_get_blocked_hosts_status(config_fs, system_data) {
const system_config_dir_version = system_data.config_directory?.config_dir_version;
const hosts_data = config_fs.get_hosts_data(system_data);
let res;
for (const host_name of Object.keys(hosts_data)) {
const host_data = hosts_data[host_name];
let version_compare_err;
const only_config_dir_upgraded = !host_data.config_dir_version && system_config_dir_version;
const only_host_upgraded = host_data.config_dir_version && !system_config_dir_version;
if (only_config_dir_upgraded) {
version_compare_err = `host's config_dir_version is undefined, system's config_dir_version already upgraded to ${system_config_dir_version}, updates to the config directory via the host will result with invalid config_dir files until the host source code upgrade`;
} else if (only_host_upgraded) {
version_compare_err = `host's config_dir_version is ${host_data.config_dir_version}, system's config_dir_version is undefined, updates to the config directory will be blocked until the config dir upgrade`;
} else {
version_compare_err = host_data.config_dir_version && system_config_dir_version &&
config_fs.compare_host_and_config_dir_version(host_data.config_dir_version, system_config_dir_version);
}
if (version_compare_err !== undefined) {
res = Object.assign(res || {}, {
[host_name]: {
host_version: host_data.current_version,
host_config_dir_version: host_data.config_dir_version,
error: version_compare_err
}
});
}
}
return res;
}

/**
* _calc_health_status calcs the overall health status of NooBaa NC
* @param {{service_status: String,
Expand Down
Loading

0 comments on commit 55d3c09

Please sign in to comment.