[Decouple] Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins #4710

manasvinibs · 2023-08-09T23:49:10Z

Description

Adds new core server service cross_compatibility_service which export feature that plugins can extend to decide what they want to do if their OpenSearch plugin counterpart is not installed or has incompatible version.

Issues Resolved

#4903
https://github.com/orgs/opensearch-project/projects/63/views/27?pane=issue&itemId=34093291

Check List

codecov · 2023-08-10T00:08:57Z

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (f822702) 66.96% compared to head (9c90d18) 66.98%.

Files	Patch %	Lines
...cross_compatibility/cross_compatibility_service.ts	90.32%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4710      +/-   ##
==========================================
+ Coverage   66.96%   66.98%   +0.01%     
==========================================
  Files        3291     3293       +2     
  Lines       63249    63281      +32     
  Branches    10057    10061       +4     
==========================================
+ Hits        42355    42386      +31     
- Misses      18453    18499      +46     
+ Partials     2441     2396      -45

Flag	Coverage Δ
Linux_1	`35.24% <ø> (ø)`
Linux_2	`55.21% <93.33%> (+0.05%)`	⬆️
Linux_3	`43.77% <8.88%> (-0.05%)`	⬇️
Linux_4	`35.33% <8.88%> (-0.03%)`	⬇️
Windows_1	`35.26% <ø> (ø)`
Windows_2	`55.18% <93.33%> (+0.05%)`	⬆️
Windows_3	`43.79% <8.88%> (-0.04%)`	⬇️
Windows_4	`35.33% <8.88%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/core/public/plugins/plugin.test.ts

src/core/server/cross_compatibility/README.md

AMoo-Miki · 2023-09-21T20:53:31Z

src/core/server/cross_compatibility/README.md

+
+The cross compatibility service can be used by plugins to:
+
+* Disable themselves if they are not compatible with the installed OpenSearch plugins.


In the above example, can you please add a condition saying if mustHavePlug is not compatble and show how I can disable my plugin?

Sure! In current world plugins can use core/capabilities API to configure whether to enable/disable plugin. I've included the example.

src/core/server/cross_compatibility/cross_compatibility_service.ts

AMoo-Miki · 2023-09-21T21:06:25Z

src/core/server/cross_compatibility/cross_compatibility_service.ts

+    opensearchInstalledPlugins.forEach((obj) => {
+      if (obj.component === depPluginName && obj.version) {
+        installedPluginVersions.add(obj.version);
+        if (semver.satisfies(semver.coerce(obj.version)!.version, depPluginVersionRange)) {


FYI, I was going to release a PEP 440 alternate to semver but I found @renovatebot/pep440 on NPM which I have not used. If that library works, it would be better solution to semver.coerce and if it doesn't, I would release my alternative and we can use it here. If you have the time, can you check if @renovatebot/pep440 works?

src/core/server/plugins/plugins_system.test.ts

src/core/types/cross_compatibility.ts

AMoo-Miki · 2023-09-21T21:14:55Z

src/core/server/plugins/plugins_system.ts

-    } catch (error) {
-      this.log.warn(
-        `Cat API call to OpenSearch to get list of plugins installed on the cluster has failed: ${error}`
+      await this.crossCompatibilityService.checkPluginVersionCompatibility(


So the health check is not doing anything here other than recording the compatibility for itself. What are the plans to notify the OSD plugins if one of their deps becomes incompatible?

Yes that's correct. To notify the OSD plugins if one of their deps becomes incompatible during runtime needs a hook to be identified to embed this new service when engine state changes. For now, plugins can call the cross compatibility service API on their own during runtime to get to know the compatibility state.

Let's create an issue for an enhancement which will have plugins possibly expose a function that we call when compatibility changes.

SuZhou-Joe · 2023-10-12T09:04:51Z

src/core/server/cross_compatibility/cross_compatibility_service.ts

+    return results;
+  }
+
+  private async isVersionCompatibleOSPluginInstalled(


The async signature seems not used in this case, could we remove the async and make the call to this method without await? As I thought there is a async call in every for of loop in the code above when there is an await.

Hi @SuZhou-Joe Thanks for your review. Yeah I don't think I need async here and in another public function. Removed the unused async usages.

src/core/server/cross_compatibility/cross_compatibility_service.ts

joshuarrrr · 2023-11-07T23:00:21Z

@mana, this has conflicts to resolve. Are you still actively working, or should it move to draft status?

manasvinibs · 2023-11-08T00:29:48Z

@mana, this has conflicts to resolve. Are you still actively working, or should it move to draft status?

Yes I can resolve the conflict. This needs secondary reviewer and I've asked @kavilla for that.

kavilla

This will be really helpful

…lugin and verify if it is installed on the cluster Signed-off-by: Manasvini B Suryanarayana <[email protected]>

…bility with OpenSearch plugins Signed-off-by: Manasvini B Suryanarayana <[email protected]>

Signed-off-by: Manasvini B Suryanarayana <[email protected]>

opensearch-trigger-bot · 2023-11-15T01:39:05Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-4710-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 cf4f9296c0f3f5dfa96df59f6efada868e7cdffe
# Push it to GitHub
git push --set-upstream origin backport/backport-4710-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-4710-to-2.x.

…gin compatibility with OpenSearch plugins (opensearch-project#4710) * Allow plugin manifest config to define semver compatible OpenSearch plugin and verify if it is installed on the cluster Signed-off-by: Manasvini B Suryanarayana <[email protected]> * Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins Signed-off-by: Manasvini B Suryanarayana <[email protected]> * Add readme doc for cross compatibility service Signed-off-by: Manasvini B Suryanarayana <[email protected]> --------- Signed-off-by: Manasvini B Suryanarayana <[email protected]> (cherry picked from commit cf4f929)

…gin compatibility with OpenSearch plugins (#4710) (#5503) * Allow plugin manifest config to define semver compatible OpenSearch plugin and verify if it is installed on the cluster * Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins * Add readme doc for cross compatibility service --------- Signed-off-by: Manasvini B Suryanarayana <[email protected]> (cherry picked from commit cf4f929)

manasvinibs requested review from ananzh, kavilla, seanneumann, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, ZilongX, Flyingliuhub and BSFishy as code owners August 9, 2023 23:49

github-actions bot added the all-star-contributor label Aug 9, 2023

manasvinibs changed the title ~~Decouple ver compat service~~ [Decouple] Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins Aug 9, 2023

manasvinibs force-pushed the decouple-verCompat-service branch from d268eff to 6a6a398 Compare August 9, 2023 23:56

manasvinibs force-pushed the decouple-verCompat-service branch from 6a6a398 to 8133a51 Compare August 10, 2023 03:49

AMoo-Miki reviewed Aug 10, 2023

View reviewed changes

src/core/public/plugins/plugin.test.ts Outdated Show resolved Hide resolved

manasvinibs force-pushed the decouple-verCompat-service branch from 8133a51 to 42d93a3 Compare August 30, 2023 22:09

manasvinibs requested a review from curq as a code owner August 30, 2023 22:09

manasvinibs force-pushed the decouple-verCompat-service branch from 42d93a3 to f341b31 Compare September 5, 2023 21:40

manasvinibs requested a review from AMoo-Miki September 5, 2023 21:40

manasvinibs force-pushed the decouple-verCompat-service branch from f341b31 to 7cfc13e Compare September 12, 2023 20:26

manasvinibs added backport 2.x v2.11.0 enhancement New feature or request labels Sep 12, 2023

ananzh assigned kavilla Sep 12, 2023

AMoo-Miki reviewed Sep 21, 2023

View reviewed changes

manasvinibs force-pushed the decouple-verCompat-service branch from 7cfc13e to 2e0179d Compare September 25, 2023 22:35

manasvinibs force-pushed the decouple-verCompat-service branch from 9a4e15a to 32ad48a Compare October 11, 2023 19:31

manasvinibs mentioned this pull request Oct 11, 2023

[Decouple] Notify OSD plugins if cluster state changes affecting the compatibility with their dependencies during runtime #5279

Open

manasvinibs force-pushed the decouple-verCompat-service branch from 32ad48a to 775c2b3 Compare October 11, 2023 20:59

manasvinibs requested a review from AMoo-Miki October 11, 2023 21:55

SuZhou-Joe reviewed Oct 12, 2023

View reviewed changes

src/core/server/cross_compatibility/cross_compatibility_service.ts Outdated Show resolved Hide resolved

SuZhou-Joe reviewed Oct 12, 2023

View reviewed changes

src/core/server/cross_compatibility/cross_compatibility_service.ts Outdated Show resolved Hide resolved

manasvinibs force-pushed the decouple-verCompat-service branch from 775c2b3 to 6c17b6b Compare October 12, 2023 18:06

manasvinibs requested a review from SuZhou-Joe October 12, 2023 18:14

manasvinibs force-pushed the decouple-verCompat-service branch from 6c17b6b to 16be1b8 Compare October 12, 2023 22:58

SuZhou-Joe approved these changes Oct 13, 2023

View reviewed changes

manasvinibs force-pushed the decouple-verCompat-service branch from 16be1b8 to 7360e2b Compare October 13, 2023 17:48

kavilla previously approved these changes Nov 14, 2023

View reviewed changes

manasvinibs added 3 commits November 14, 2023 18:59

Allow plugin manifest config to define semver compatible OpenSearch p…

aeccac4

…lugin and verify if it is installed on the cluster Signed-off-by: Manasvini B Suryanarayana <[email protected]>

Add new core service to expose functionality to verify plugin compati…

ad1f3e3

…bility with OpenSearch plugins Signed-off-by: Manasvini B Suryanarayana <[email protected]>

Add readme doc for cross compatibility service

9c90d18

Signed-off-by: Manasvini B Suryanarayana <[email protected]>

manasvinibs dismissed kavilla’s stale review via 9c90d18 November 14, 2023 19:09

manasvinibs force-pushed the decouple-verCompat-service branch from 7360e2b to 9c90d18 Compare November 14, 2023 19:09

AMoo-Miki approved these changes Nov 15, 2023

View reviewed changes

kavilla approved these changes Nov 15, 2023

View reviewed changes

manasvinibs merged commit cf4f929 into opensearch-project:main Nov 15, 2023
67 checks passed

opensearch-trigger-bot bot added the failed backport label Nov 15, 2023

manasvinibs mentioned this pull request Nov 17, 2023

[Manual][Backport 2.x][Decouple] Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins #5503

Merged

7 tasks

manasvinibs removed the failed backport label Jan 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Decouple] Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins #4710

[Decouple] Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins #4710

manasvinibs commented Aug 9, 2023 •

edited

Loading

codecov bot commented Aug 10, 2023 •

edited

Loading

AMoo-Miki Sep 21, 2023

manasvinibs Sep 26, 2023

AMoo-Miki Sep 21, 2023

AMoo-Miki Sep 21, 2023

manasvinibs Sep 25, 2023

AMoo-Miki Oct 3, 2023

manasvinibs Oct 11, 2023

SuZhou-Joe Oct 12, 2023 •

edited

Loading

manasvinibs Oct 12, 2023

joshuarrrr commented Nov 7, 2023

manasvinibs commented Nov 8, 2023

kavilla left a comment

opensearch-trigger-bot bot commented Nov 15, 2023


		The cross compatibility service can be used by plugins to:

		* Disable themselves if they are not compatible with the installed OpenSearch plugins.

[Decouple] Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins #4710

[Decouple] Add new core service to expose functionality to verify plugin compatibility with OpenSearch plugins #4710

Conversation

manasvinibs commented Aug 9, 2023 • edited Loading

Description

Issues Resolved

Check List

codecov bot commented Aug 10, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SuZhou-Joe Oct 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshuarrrr commented Nov 7, 2023

manasvinibs commented Nov 8, 2023

kavilla left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Nov 15, 2023

manasvinibs commented Aug 9, 2023 •

edited

Loading

codecov bot commented Aug 10, 2023 •

edited

Loading

SuZhou-Joe Oct 12, 2023 •

edited

Loading