Blog post: Backwards Compatibility Testing for OpenSearch #419
Conversation
Signed-off-by: Vacha <[email protected]>
|
@ahopp was not able to add you as a reviewer, so tagging you here. |
|
Hey @VachaShah thanks for putting this post together! Is there a linked issue? If not, can you open one using this template to help @stockholmux and I queue this up? |
Hey @elfisher, just created an issue for this #428. Thank you! |
There was a problem hiding this comment.
This is a meaty post, I like it.
- Change links to GitHub to permalinks and introduce the code earlier. In the parts of the framework and test cluster setup you talk about a gradle plugin and instances of classes, link everything.
- Add examples of what versions are being tested as part of 1.1 and 1.2 releases, and what's done automatically and what is not.
- Capitalize Section Titles Consistently
- Post needs a careful grammar pass.
|
|
||
| ## Test cluster setup | ||
|
|
||
| The test clusters are available via `opensearch.testclusters` gradle plugin. A cluster is an instance of `OpenSearchCluster` and it contains a list of nodes which are instances of the `OpenSearchNode` class which provides the methods to start, stop, restart, upgrade, install plugins etc. The version of the nodes can be set by providing a list of versions to the `testclusters`. The node can then be upgraded to a newer version provided by using the `nextNodeToNextVersion` method on the concerned testcluster or by using `goToNextVersion` method which upgrades all the nodes in a cluster. |
There was a problem hiding this comment.
Link/code pointer to the gradle plugin would be awesome.
There was a problem hiding this comment.
'concerned testcluster or by using' (add back ticks for testcluster.
|
|
||
| ## Summary | ||
|
|
||
| Backwards compatibility specially for plugins has been a major concern during releases since it involved manual testing to determine safe upgrade paths. With the extended framework for BWC, plugins can now implement their BWC tests thus alleviating the need for manual efforts. All pull requests in OpenSearch now run the BWC tests making sure all the changes adhere to the backwards compatibility standards. |
There was a problem hiding this comment.
Link to an example CI would be nice.
|
|
||
| For more information on BWC testing, please check out [OpenSearch/TESTING.md](https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-backwards-compatibility) and [opensearch-plugins/TESTING.md](https://github.com/opensearch-project/opensearch-plugins/blob/main/TESTING.md#backwards-compatibility-testing). | ||
|
|
||
| Closing on a high note, we hope this blog post adds light to the backwards compatibility framework and we are continuing to invest efforts in automating backwards compatibility for OpenSearch. |
There was a problem hiding this comment.
May be link the issue opensearch-project/opensearch-build#90 as next steps?
There was a problem hiding this comment.
Really nice post @VachaShah and @saratvemulapalli. It clear and makes sense - just a few little points of grammar and disfluency.
| description: "This post provides details on the framework in OpenSearch for backwards compatibility testing" | ||
| --- | ||
|
|
||
| Backwards Compatibility (BWC) testing is used to test and determine the safe upgrade paths from a supported BWC version to the current version. The framework in OpenSearch allows to run these BWC tests for all supported BWC versions in OpenSearch allowing safe upgrade paths between versions. This framework is now extended to work for plugins and they can introduce their BWC tests without creating individual frameworks of their own. This post provides details on the framework in OpenSearch for backwards compatibility. |
There was a problem hiding this comment.
'framework in OpenSearch allows you to run these BWC tests'
'work for plugins which can introduce their BWC tests without creating individual frameworks'
|
|
||
| ## The Framework | ||
|
|
||
| As a general idea for BWC tests, the framework supports to spin up a test cluster with a supported BWC version, then upgrade the nodes to the current version and test various features and functionalities as a result of the upgrade. This allows to test the compatibility of the code between versions. |
There was a problem hiding this comment.
'the framework supports spinning up a test cluster'
'This allows for testing the'
|
|
||
| ## Test cluster setup | ||
|
|
||
| The test clusters are available via `opensearch.testclusters` gradle plugin. A cluster is an instance of `OpenSearchCluster` and it contains a list of nodes which are instances of the `OpenSearchNode` class which provides the methods to start, stop, restart, upgrade, install plugins etc. The version of the nodes can be set by providing a list of versions to the `testclusters`. The node can then be upgraded to a newer version provided by using the `nextNodeToNextVersion` method on the concerned testcluster or by using `goToNextVersion` method which upgrades all the nodes in a cluster. |
There was a problem hiding this comment.
'concerned testcluster or by using' (add back ticks for testcluster.
|
|
||
| The test clusters are available via `opensearch.testclusters` gradle plugin. A cluster is an instance of `OpenSearchCluster` and it contains a list of nodes which are instances of the `OpenSearchNode` class which provides the methods to start, stop, restart, upgrade, install plugins etc. The version of the nodes can be set by providing a list of versions to the `testclusters`. The node can then be upgraded to a newer version provided by using the `nextNodeToNextVersion` method on the concerned testcluster or by using `goToNextVersion` method which upgrades all the nodes in a cluster. | ||
|
|
||
| An example testcluster initialization is given below. Here the testcluster is named `mycluster` with 4 nodes. Two versions are provided and the first version is the version that gets installed by default when a node starts. It upgrades to the next version using the method mentioned above. |
There was a problem hiding this comment.
back ticks around testcluster
| * **Rolling Upgrade:** For a rolling upgrade scenario, a test cluster is spun up with multiple nodes of a supported BWC version and each node is upgraded one by one to a newer version of OpenSearch using the `nextNodeToNextVersion` method resulting in a fulling upgraded cluster achieved via sequential upgrade. The corresponding tests are run for the old cluster and for each node after it undergoes the upgrade. | ||
| * Located in `qa/rolling-upgrade` | ||
| * Run using `./gradlew :qa:rolling-upgrade:vM.N.b#upgradedClusterTest` | ||
| * **Full Cluster Restart:** For a full cluster restart, a test cluster is spun up with multiple nodes of a supported BWC version and all the nodes are upgraded using the [`goToNextVersion`](https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java#L388) method resulting in a fully upgrade cluster by restarting. The corresponding tests are run for the old cluster and for the upgraded cluster. |
There was a problem hiding this comment.
'fully upgraded cluster by restarting'
| * Run using `./gradlew :qa:repository-multi-version:vM.N.b#Step4NewClusterTest` | ||
|
|
||
| To run all the backwards compatibility tests: | ||
| `./gradlew bwcTest` |
There was a problem hiding this comment.
Add another line break after the code block - it runs together into one paragraph on the site.
|
|
||
| ## Summary | ||
|
|
||
| Backwards compatibility specially for plugins has been a major concern during releases since it involved manual testing to determine safe upgrade paths. With the extended framework for BWC, plugins can now implement their BWC tests thus alleviating the need for manual efforts. All pull requests in OpenSearch now run the BWC tests making sure all the changes adhere to the backwards compatibility standards. |
There was a problem hiding this comment.
'Backwards compatibility especially for plugins'
| } | ||
| ``` | ||
|
|
||
| Mixed cluster, rolling upgrade and full restart upgrade scenarios are supported for the BWC tests in plugins and the nodes in the cluster can be upgraded using the methods on the testcluster: [`upgradeNodeAndPluginToNextVersion`](https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java#L413) for a single node upgrade at a time and [`upgradeAllNodesAndPluginsToNextVersion`](https://github.com/opensearch-project/OpenSearch/blob/main/buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchCluster.java#L395) for upgrading all the nodes at the same time. The tests corresponding to each step can be configured to run similar to explained above according to each upgrade scenario. |
There was a problem hiding this comment.
'configured to run similarly as explained above'
Just realized @VachaShah added me as an author. She did all the work so credits go to her. |
Signed-off-by: Vacha <[email protected]>
|
I'm satisfied with the post - as soon as @dblock and @saratvemulapalli give the nod, we can publish. |
|
Looks good to me, ship it!! |
|
@dblock Any objections? |
There was a problem hiding this comment.
This is good! We can publish this or consider some potential updates before we do:
- I think we can say more about which versions we test bcw with more explicitly in the intro. Maybe "For example, OpenSearch 1.1.0 was tested for backwards compatibility for versions ...".
- We have some execution framework in openseach-build that runs bcw tests, but it's not perfect. You could expand on how and when we'd like to run bcw tests and link related issues other than opensearch-build#90.
| categories: | ||
| - technical-post | ||
| twittercard: | ||
| description: "This post provides details on the framework in OpenSearch for backwards compatibility testing" |
There was a problem hiding this comment.
Missing period, it's a full sentence.
There was a problem hiding this comment.
@VachaShah Can you address @dblock's points above before lunch today (2021-11-05)? I'd love to get this out the door today.
There was a problem hiding this comment.
@stockholmux , @VachaShah is out today.
I don't have enough permissions to make changes on her fork.
May be can publish this and then tweak it next week?
There was a problem hiding this comment.
@saratvemulapalli OK. Let's do that!
There was a problem hiding this comment.
Thank you @stockholmux @saratvemulapalli @dblock ! I will put up a PR for the tweak this week.
Signed-off-by: Vacha [email protected]
Description
Blog post for Backwards Compatibility Testing for OpenSearch.
Issues Resolved
Closes #428
By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.