add support for s390x architecture

[pantovad]

Is your feature request related to a problem? Please describe.
OpenSearch does not support IBM s390x platform which is understandable. However, with some slight changes we manage to run it successfully on IBM LinuxONE and benefit from impressive performance this platform provides. The only issue preventing us to have the OpenSearch server fully portable without the need to do additional changes in Gradle and rebuild the server is the lack of support for s390x in SystemCallFilter (a single line in the ARCHITECTURES hash map).

Describe the solution you'd like
I would happily provide the missing entry in the hash as a PR.

Describe alternatives you've considered
none

Additional context
This should be a simple and safe change and it should not cause any regressions for the supported platforms yet it would help us significantly in our deployment process.

[andrross]

Thanks for the contribution!

@dblock Do you have any thoughts here?

[pantovad]

FYI the OpenSearch/OpenSearch Dashboards Docker images for s390x are publicly available at:

• https://hub.docker.com/repository/docker/phoenixsystemsag/opensearch
• https://hub.docker.com/repository/docker/phoenixsystemsag/opensearch-dashboards

The only missing feature is under Dashboards Reporting plugin (PDF/PNG reports) as no Chrome/Chromium available for s390x (used by Puppeteer NodeJS module).

[andrross]

@dbwiddis Can you take a look as well? I think this might be relevant to some of the things you've been looking at.

[dbwiddis]

@andrross thanks for the tag. I have been looking into adding support for other architectures, see #1303 and #2341. Currently blocked by conditionals in the build scripts that assume only two architectures, that I haven't had time to dig into, see draft PR I'm spending a few spare cycles on in #3930.

@pantovad have you been able to build locally on s390x architecture without any other changes than this one?

[pantovad]

@dbwiddis sure I had to patch Gradle setup. It was a minimal effort and dirty approach just to get the server.jar built (due to SystemCall Filter change - see patch below for v1.3.2).
The rest of the OpenSearch artifacts were portable (thanks to pure Java and JNA) with exceptions (JNI used but no s390x native libs available):

• sqlite-jdbc in performance-analyzer-rca - had to build - available at https://github.com/pantovad/sqlite-jdbc/releases/download/3.32.3.2/sqlite-jdbc-3.32.3.2.jar
• k-NN plugin - again had to build for s390x native libs

index 38d6db8..8104682 100644
--- a/buildSrc/src/main/java/org/opensearch/gradle/Architecture.java
+++ b/buildSrc/src/main/java/org/opensearch/gradle/Architecture.java
@@ -35,7 +35,8 @@ package org.opensearch.gradle;
 public enum Architecture {
 
     X64,
-    ARM64;
+    ARM64,
+    S390X;
 
     public static Architecture current() {
         final String architecture = System.getProperty("os.arch", "");
@@ -45,6 +46,8 @@ public enum Architecture {
                 return X64;
             case "aarch64":
                 return ARM64;
+           case "s390x":
+                return S390X;
             default:
                 throw new IllegalArgumentException("can not determine architecture from [" + architecture + "]");
         }
diff --git a/buildSrc/src/main/java/org/opensearch/gradle/Jdk.java b/buildSrc/src/main/java/org/opensearch/gradle/Jdk.java
index debac7b..c52ba83 100644
--- a/buildSrc/src/main/java/org/opensearch/gradle/Jdk.java
+++ b/buildSrc/src/main/java/org/opensearch/gradle/Jdk.java
@@ -48,7 +48,7 @@ import java.util.regex.Pattern;
 
 public class Jdk implements Buildable, Iterable<File> {
 
-    private static final List<String> ALLOWED_ARCHITECTURES = Collections.unmodifiableList(Arrays.asList("aarch64", "x64"));
+    private static final List<String> ALLOWED_ARCHITECTURES = Collections.unmodifiableList(Arrays.asList("aarch64", "x64", "s390x"));
     private static final List<String> ALLOWED_VENDORS = Collections.unmodifiableList(Arrays.asList("adoptium", "adoptopenjdk", "openjdk"));
     private static final List<String> ALLOWED_PLATFORMS = Collections.unmodifiableList(
         Arrays.asList("darwin", "freebsd", "linux", "mac", "windows")
diff --git a/distribution/archives/build.gradle b/distribution/archives/build.gradle
index 2c5b91f..1d21bc8 100644
--- a/distribution/archives/build.gradle
+++ b/distribution/archives/build.gradle
@@ -137,6 +137,13 @@ distribution_archives {
     }
   }
 
+  linuxS390xTar {
+    archiveClassifier = 'linux-s390x'
+    content {
+      archiveFiles(modulesFiles('linux-s390x'), 'tar', 'linux', 's390x', false)
+    }
+  }
+
   windowsZip {
     archiveClassifier = 'windows-x64'
     content {
diff --git a/distribution/build.gradle b/distribution/build.gradle
index 73d90af..c181347 100644
--- a/distribution/build.gradle
+++ b/distribution/build.gradle
@@ -252,7 +252,7 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
   // Setup all required JDKs
   project.jdks {
     ['darwin', 'linux', 'windows'].each { platform ->
-      (platform == 'linux' ? ['x64', 'aarch64'] : ['x64']).each { architecture ->
+      (platform == 'linux' ? ['x64', 'aarch64', 's390x'] : ['x64']).each { architecture ->
         "bundled_${platform}_${architecture}" {
           it.platform = platform
           it.version = VersionProperties.getBundledJdk(platform)
@@ -325,7 +325,7 @@ configure(subprojects.findAll { ['archives', 'packages'].contains(it.name) }) {
           }
         }
         def buildModules = buildModulesTaskProvider
-        List excludePlatforms = ['darwin-x64', 'freebsd-x64', 'linux-x64', 'linux-arm64', 'windows-x64']
+        List excludePlatforms = ['darwin-x64', 'freebsd-x64', 'linux-x64', 'linux-arm64', 'linux-s390x', 'windows-x64']
         if (platform != null) {
           excludePlatforms.remove(excludePlatforms.indexOf(platform))
         } else {
diff --git a/server/src/main/java/org/opensearch/bootstrap/SystemCallFilter.java b/server/src/main/java/org/opensearch/bootstrap/SystemCallFilter.java
index 8e179de..5521d11 100644
--- a/server/src/main/java/org/opensearch/bootstrap/SystemCallFilter.java
+++ b/server/src/main/java/org/opensearch/bootstrap/SystemCallFilter.java
@@ -257,6 +257,7 @@ final class SystemCallFilter {
         Map<String, Arch> m = new HashMap<>();
         m.put("amd64", new Arch(0xC000003E, 0x3FFFFFFF, 57, 58, 59, 322, 317));
         m.put("aarch64", new Arch(0xC00000B7, 0xFFFFFFFF, 1079, 1071, 221, 281, 277));
+        m.put("s390x", new Arch(0x80000016, 0xFFFFFFFF, 2, 190, 11, 354, 348));
         ARCHITECTURES = Collections.unmodifiableMap(m);
     }
 
diff --git a/settings.gradle b/settings.gradle
index 79edc28..117e8cd 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -43,13 +43,14 @@ List projects = [
   'distribution:archives:freebsd-tar',
   'distribution:archives:no-jdk-freebsd-tar',
   'distribution:archives:linux-arm64-tar',
+  'distribution:archives:linux-s390x-tar',
   'distribution:archives:linux-tar',
   'distribution:archives:no-jdk-linux-tar',
-  'distribution:docker',
-  'distribution:docker:docker-arm64-build-context',
-  'distribution:docker:docker-arm64-export',
-  'distribution:docker:docker-build-context',
-  'distribution:docker:docker-export',
+//  'distribution:docker',
+//  'distribution:docker:docker-arm64-build-context',
+//  'distribution:docker:docker-arm64-export',
+//  'distribution:docker:docker-build-context',
+//  'distribution:docker:docker-export',
   'distribution:packages:arm64-deb',
   'distribution:packages:deb',
   'distribution:packages:no-jdk-deb',
@@ -113,7 +114,7 @@ project(':example-plugins').projectDir = new File(rootProject.projectDir, 'plugi
 addSubProjects('', new File(rootProject.projectDir, 'libs'))
 addSubProjects('', new File(rootProject.projectDir, 'modules'))
 addSubProjects('', new File(rootProject.projectDir, 'plugins'))
-addSubProjects('', new File(rootProject.projectDir, 'qa'))
+//addSubProjects('', new File(rootProject.projectDir, 'qa'))
 addSubProjects('test', new File(rootProject.projectDir, 'test/external-modules'))
 
 List startTasks = gradle.startParameter.taskNames

Built with:
./gradlew --no-daemon --refresh-dependencies -DskipTests=true -Dbuild.snapshot=false -Dopensearch.version=1.3.2 :server:jar

[dbwiddis]

@pantovad Excellent! Thanks for that diff. You found all the points I hadn't yet found in my draft PR #3930.

Can i ask you to update your PR #4001 to include these build changes as well? Also could you take the test class that I wrote in #3930 and include it (remove the PPC and RISCV portions and just keep your S390X) part? I can then close #3930 and use your #4001 PR as a template for adding those other architectures later.

[pantovad]

Hi @dbwiddis sorry for late response (summer holidays :) ). I updates PR #4001 as requested. Please have a look.

[dblock]

Thanks for the contribution!

@dblock Do you have any thoughts here?

I realized that I never answered this, sorry! I am looking forward to merging #4001

[francesco-defalco]

Hi,

thank you in advance for you job, I am trying to run the image phoenixsystemsag/opensearch:2.3.0 in docker simulated for s390x but I receiving this error:

Did you run the version? If in your case run correctly which is your configuration?

Thank you

[dblock]

I believe this was merged into 2.4.0, but didn't make it into 2.3.0 (released Sep 13). I don't see it in https://github.com/opensearch-project/OpenSearch/commits/2.3.

[dblock]

@peterzhuamazon What's the docker image version that @francescodefalco can try?

[peterzhuamazon]

opensearchstaging/opensearch:2.4.0

[francesco-defalco]

opensearchstaging/opensearch:2.4.0

Hi @pantovad, from what I see there is not s390x support for that image.
Do I need to do some particular step?

Thank you,

EDIT: I realized that I mentioned the dashboard image, instead I mean the opensearch base image that Is not possible to build for s390x platform (with buildx option).

[dblock]

I think "support for s390x" means more than just this change here. I've opened opensearch-project/opensearch-build#2836 as an umbrella and to track all child items and will close this one since support was added to the engine itself. Let's take a discussion there?

[pantovad]

opensearchstaging/opensearch:2.4.0

Hi @pantovad, from what I see there is not s390x support for that image. Do I need to do some particular step?

Thank you,

EDIT: I realized that I mentioned the dashboard image, instead I mean the opensearch base image that Is not possible to build for s390x platform (with buildx option).

See my comments above in this issue - Opensearch Dashboards require additional changes in order to have the build process running on s390x (native modules used during the build process) - what I did instead is cloning the amd64 image and just rebuilding the native runtime nodeJS modules ([email protected]) - images publicly available at https://hub.docker.com/repository/docker/phoenixsystemsag/opensearch-dashboards