[SPARK-27178][k8s] add nss to the spark/k8s Dockerfile

[shaneknapp]

What changes were proposed in this pull request?

while performing some tests on our existing minikube and k8s infrastructure, i noticed that the integration tests were failing. i dug in and discovered the following message buried at the end of the stacktrace:

  Caused by: java.io.FileNotFoundException: /usr/lib/libnss3.so
  	at sun.security.pkcs11.Secmod.initialize(Secmod.java:193)
  	at sun.security.pkcs11.SunPKCS11.<init>(SunPKCS11.java:218)
  	... 81 more

after i added the nss package to resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile, everything worked.

this is also impacting current builds. see: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8959/console

How was this patch tested?

i tested locally before pushing, and the build system will test the rest.

[SparkQA]

Test build #103556 has started for PR 24111 at commit 48e7a0c.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8967/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8967/

[shaneknapp]

test this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8969/

[SparkQA]

Test build #103558 has finished for PR 24111 at commit 48e7a0c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[erikerlandson]

Was this introduced by some other dependency version bump?

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8969/

[shaneknapp]

Was this introduced by some other dependency version bump?

the only other PR merged that touched k8s after the client bump on wednesday was this one:
#23380

...and that doesn't touch anything i care about. :\

since i don't want to ruin my weekend, i will investigate further next week.

[felixcheung]

sun.security - is it running with a new JDK?

[srowen]

Seems OK to me, but I don't know much about this.

[shaneknapp]

sun.security - is it running with a new JDK?

i dunno... it's in the docker container, which is pulling upstream from alpine linux's dockerhub repo.

[markhamstra]

Seems that there is an argument to be made that the Dockerfile should lock down a much more specific version of Alpine and JDK. Using what is essentially a floating tag leaves us guessing any time these dependencies change.

[shaneknapp]

Seems that there is an argument to be made that the Dockerfile should lock down a much more specific version of Alpine and JDK. Using what is essentially a floating tag leaves us guessing any time these dependencies change.

yep, i agree completely. it's not at all where we want to be.

three things i will do tomorrow:

1. open a jira to discuss locking any dockerfiles to a specific version (maybe host our own?)

2. open a new PR for 2.4.x w/this fix

3. looking in to any potential dep changes in the image

sorry, four thing:

4. pick up a bottle of bourbon to help with (3)

:)

[attilapiros]

ping @vanzin

[vanzin]

Actually I wonder if this has been fixed already upstream. Since both the jdk and the base libs come from the base image, it would be a bug in the base image if this library were missing.

With the latest openjdk:8-alpine:

$ docker run -it openjdk:8-alpine
/ # java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (IcedTea 3.10.0) (Alpine 8.191.12-r0)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
/ # ls -l /usr/lib/libnss3.so
lrwxrwxrwx    1 root     root            13 Mar  8 02:13 /usr/lib/libnss3.so -> libnss3.so.41

Not sure if the IT scripts are forcing the image to refresh or use a cached version, though...

[shaneknapp]

Actually I wonder if this has been fixed already upstream. Since both the jdk and the base libs come from the base image, it would be a bug in the base image if this library were missing.

With the latest openjdk:8-alpine:

$ docker run -it openjdk:8-alpine
/ # java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (IcedTea 3.10.0) (Alpine 8.191.12-r0)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
/ # ls -l /usr/lib/libnss3.so
lrwxrwxrwx    1 root     root            13 Mar  8 02:13 /usr/lib/libnss3.so -> libnss3.so.41

Not sure if the IT scripts are forcing the image to refresh or use a cached version, though...

hmm, lemme remove the nss package install and re-run the k8s integration tests locally and see what happens.

[shaneknapp]

ok... so the last time this image on dockerhub was updated was march 1st, which doesn't fit in to our timing of these failures happening. i'm still mildly confused.

i'm also wondering if this is something that's minikube-specific... since we're using the convoluted path of minikube -> kvm2 -> docker-machine things may not be behaving as expected.

also, i re-tested the integration tests w/nss removed from apk add and the jobs are failing "as expected" and passing w/nss added to the Dockerfile.

[vanzin]

that's weird because "apk info nss" on the bare image shows that it is installed, so this change should be a no op.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
openjdk             8-alpine            e9ea51023687        10 days ago         105MB

[vanzin]

Ah, I think this is it:

 ---> Running in 23e86476ab1c
+ apk upgrade --no-cache
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/7) Upgrading libcrypto1.1 (1.1.1a-r1 -> 1.1.1b-r1)
(2/7) Upgrading libssl1.1 (1.1.1a-r1 -> 1.1.1b-r1)
(3/7) Upgrading openjdk8-jre-base (8.191.12-r0 -> 8.201.08-r0)
(4/7) Upgrading openjdk8-jre (8.191.12-r0 -> 8.201.08-r0)
(5/7) Purging nss (3.41-r0)

Now why the upgrade is purging the nss package, I have no idea... seems like a bug in some alpine package (probably the updated openjre 8.201.08-r0), but this seems ok as a workaround.

[shaneknapp]

ok cool. i'll test this one more time and then merge it to master. 2.4.1 will be it's own PR.

[shaneknapp]

test this please

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/9034/

[SparkQA]

Test build #103633 has finished for PR 24111 at commit 48e7a0c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/9034/

[shaneknapp]

alright, merging to master.

[shaneknapp]

actually i'm going to wait for a couple of hours just to be sure my access to repos is properly synced.