[BUG] OpenSearch Core 1.3.0 including ja in JDK man pages

[peterzhuamazon]

In rpm assemble through build repo code, we will extract the files from core rpm.

It failed with this error once we switch from 14 to 11 on JDK for 1.3.0.

cpio: cannot make directory `./usr/share/opensearch/jdk/man/ja': No such file or directory
cpio: ./usr/share/opensearch/jdk/man/ja/man1: Cannot mkdir: No such file or directory

Root cause:

jdk14:
opensearch-1.2.4/jdk/man
drwxr-x--- 2 <> 4096 Jan 14 03:42 man1

jdk11:
opensearch-1.3.0/jdk/man
lrwxrwxrwx 1 <>   11 Mar 15 19:28 ja -> ja_JP.UTF-8
drwxr-x--- 3 <> 4096 Jan  1  1970 ja_JP.UTF-8
drwxr-x--- 2 <> 4096 Jan  1  1970 man1

There is a new folder ja_JP.UTF-8 and a new symlink ja which caused cpio to fail as it tries to treat symlink as a folder here.

General question on why this folder is needed as a result.
And whether we need to make changes on cpio to omit this error.

Thanks.

[peterzhuamazon]

Update: This is only for jdk11 in 1.3.0. As for jdk17 in 2.0.0 this is not an issue at all

total 4
drwxr-xr-x 2 zhujiaxi amazon 4096 Mar 21 19:41 man1

[bbarani]

@CEHENKLE @saratvemulapalli This is blocking RPM builds. Can you please prioritize it?

[dblock]

How does one reproduce this? Where's the cpio code?

[saratvemulapalli]

@peterzhuamazon could you help understand what process is cpio doing?

From what I see JDK versions < 12 have Japan version of man pages bundled.
The ja directory is a symlink which points to the JP manpages.
This is nothing specific to OpenSearch and comes from JDK out of the box.

As far as I understand its a no-op and you should be able to skip the ja directory.
But my recommendation would be to package as JDK distributions do (i.e include ja symlink).
This will make us follow the convention rather than tearing it apart.

Ref:
[1] https://mail.openjdk.java.net/pipermail/build-dev/2020-June/027643.html

[peterzhuamazon]

Easiest way to reproduce this one:

# Build rpm
./build.sh manifests/1.3.0/opensearch-1.3.0.yml --platform linux --distribution rpm --component OpenSearch

# Try extraction
cd builds/opensearch/dist
rpm2cpio <rpm file> | cpio -imdv

You will see:

cpio: cannot make directory `./usr/share/opensearch/jdk/man/ja': No such file or directory
cpio: ./usr/share/opensearch/jdk/man/ja/man1: Cannot mkdir: No such file or directory

Then check usr/share/opensearch/jdk/man and you see jp.

This does not happen on jdk14 or jdk17, only on jdk11.

Thanks.

[peterzhuamazon]

Since cpio does not process symlink well, it will cause the cpio to return error code 2 and stop the assemble process.
If we want to release 1.3.x version of rpm then this is a blocker.
Not a blocker for 2.0.0+ for now.

[saratvemulapalli]

Since cpio does not process symlink well, it will cause the cpio to return error code 2 and stop the assemble process. If we want to release 1.3.x version of rpm then this is a blocker. Not a blocker for 2.0.0+ for now.

I understand that. As i've linked, JDK versions < 12 will have this problem. ( I've verified 8,11 not sure about others)
Anyway, whats the question here for OpenSearch?
If its the tooling problem lets figure that out.

[dblock]

https://superuser.com/questions/1005740/how-do-i-deal-with-unpacking-of-archive-failed-cpio-symlink seems to offer some options - I am not familiar with cpio to say what we should be doing about this

[peterzhuamazon]

After talking to @saratvemulapalli and sees this is only an additional symlink due to JDK pre 12 incorrectly include it.
If we are releasing 1.3.x rpm this would be blocking, we probably need to add some additional script / code so ja can be removed before the 1st bundling.
If we are releasing 2.0.x rpm this would not be an issue.

Thanks.

[saratvemulapalli]

Closing this.
Feel free to re-open @peterzhuamazon if you have more questions!

[peterzhuamazon]

@saratvemulapalli We will support 1.3.0 version of OpenSearch will need to address this with a patch in core by creating a separate branch based on tags/1.3.0 and remove ja related symlink/folder before core rpm is built.

[dblock]

@saratvemulapalli We will support 1.3.0 version of OpenSearch will need to address this with a patch in core by creating a separate branch based on tags/1.3.0 and remove ja related symlink/folder before core rpm is built.

This needs to be a 1.3.1 because it's no longer a 1.3.0.

[peterzhuamazon]

1. Long Term: Support TYPE during core build, TYPE=archives/packages/dirs.
   User can do distribution=rpm;type=dirs and they will be provided with directory of files that is built for rpm
2. Short Term: Creating a new branch of 1.3-rpm and allow core to delete ja related links/folder during build.

[saratvemulapalli]

1. I love the long term approach, keeps it simple for all our bundling. Could you open an issue for this?
   I would invest time on this as it helps our bundling with tarball, RPM and DEB for all versions down the line.
2. I dont like the short term approach because the rootcause is the tool, it doesn’t understand symlinks. The symlink was generated with JDK distribution (nothing to do with OpenSearch).
   I would rather have the rootcause fixed, than putting in a hack in OpenSearch (and be worried about maintaining it forever).

[peterzhuamazon]

@dblock @saratvemulapalli This PR #2602 is raised to resolve this issue temporarily.
@xuezhou25 is working on resolving #2571 on a separate PR.

Thanks.

[peterzhuamazon]

Testing proves that dashboards does not have similar issues:

./usr/share/opensearch-dashboards/src/setup_node_env/root/is_root.js
./var/lib/opensearch-dashboards
582913 blocks

% echo $?
0