Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-23038][TEST] Update docker/spark-test (JDK/OS) #20230

Closed
wants to merge 1 commit into from
Closed

[SPARK-23038][TEST] Update docker/spark-test (JDK/OS) #20230

wants to merge 1 commit into from

Conversation

dongjoon-hyun
Copy link
Member

What changes were proposed in this pull request?

This PR aims to update the followings in docker/spark-test.

  • JDK7 -> JDK8
    Spark 2.2+ supports JDK8 only.

  • Ubuntu 12.04.5 LTS(precise) -> Ubuntu 16.04.3 LTS(xeniel)
    The end of life of precise was April 28, 2017.

How was this patch tested?

Manual.

  • Master
$ cd external/docker
$ ./build
$ export SPARK_HOME=...
$ docker run -v $SPARK_HOME:/opt/spark spark-test-master
CONTAINER_IP=172.17.0.3
...
18/01/11 06:50:25 INFO MasterWebUI: Bound MasterWebUI to 172.17.0.3, and started at http://172.17.0.3:8080
18/01/11 06:50:25 INFO Utils: Successfully started service on port 6066.
18/01/11 06:50:25 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
18/01/11 06:50:25 INFO Master: I have been elected leader! New state: ALIVE
  • Slave
$ docker run -v $SPARK_HOME:/opt/spark spark-test-worker spark://172.17.0.3:7077
CONTAINER_IP=172.17.0.4
...
18/01/11 06:51:54 INFO Worker: Successfully registered with master spark://172.17.0.3:7077

After slave starts, master will show

18/01/11 06:51:54 INFO Master: Registering worker 172.17.0.4:8888 with 4 cores, 1024.0 MB RAM

dongjoon-hyun
dongjoon-hyun commented Jan 11, 2018
View reviewed changes
external/docker/spark-test/base/Dockerfile
# Remove unneeded /var/lib/apt/lists/* after install to reduce the
# docker image size (by ~30MB)
RUN apt-get update && \
apt-get install -y less openjdk-7-jre-headless net-tools vim-tiny sudo openssh-server && \
apt-get install -y less openjdk-8-jre-headless iproute2 vim-tiny sudo openssh-server && \
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is required to use ip command in xenial.

@SparkQA
Copy link

SparkQA commented Jan 11, 2018

Test build #85953 has finished for PR 20230 at commit cc3321c.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

srowen
srowen reviewed Jan 11, 2018
View reviewed changes
external/docker/spark-test/base/Dockerfile
@@ -15,14 +15,14 @@
# limitations under the License.
#

FROM ubuntu:precise
FROM ubuntu:xenial
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably a good change, but is it required in order to get Java 8?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It's required. When we change only jdk7 to jdk8, it fails.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea no 8 on precise

@dongjoon-hyun
Copy link
Member Author

Retest this please.

@SparkQA
Copy link

SparkQA commented Jan 11, 2018

Test build #85977 has finished for PR 20230 at commit cc3321c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Jan 12, 2018 •
edited
Loading

Hi, @aarondav .
Could you review this?

felixcheung
felixcheung approved these changes Jan 13, 2018
View reviewed changes
external/docker/spark-test/base/Dockerfile
@@ -15,14 +15,14 @@
# limitations under the License.
#

FROM ubuntu:precise
FROM ubuntu:xenial
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea no 8 on precise

@dongjoon-hyun
Copy link
Member Author

Thank you for review and approval, @felixcheung .

@asfgit asfgit closed this in 7a3d0aa Jan 14, 2018
asfgit pushed a commit that referenced this pull request Jan 14, 2018
@dongjoon-hyun
[SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
a335a49 
## What changes were proposed in this pull request?

This PR aims to update the followings in `docker/spark-test`.

- JDK7 -> JDK8
Spark 2.2+ supports JDK8 only.

- Ubuntu 12.04.5 LTS(precise) -> Ubuntu 16.04.3 LTS(xeniel)
The end of life of `precise` was April 28, 2017.

## How was this patch tested?

Manual.

* Master
```
$ cd external/docker
$ ./build
$ export SPARK_HOME=...
$ docker run -v $SPARK_HOME:/opt/spark spark-test-master
CONTAINER_IP=172.17.0.3
...
18/01/11 06:50:25 INFO MasterWebUI: Bound MasterWebUI to 172.17.0.3, and started at http://172.17.0.3:8080
18/01/11 06:50:25 INFO Utils: Successfully started service on port 6066.
18/01/11 06:50:25 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
18/01/11 06:50:25 INFO Master: I have been elected leader! New state: ALIVE
```

* Slave
```
$ docker run -v $SPARK_HOME:/opt/spark spark-test-worker spark://172.17.0.3:7077
CONTAINER_IP=172.17.0.4
...
18/01/11 06:51:54 INFO Worker: Successfully registered with master spark://172.17.0.3:7077
```

After slave starts, master will show
```
18/01/11 06:51:54 INFO Master: Registering worker 172.17.0.4:8888 with 4 cores, 1024.0 MB RAM
```

Author: Dongjoon Hyun <[email protected]>

Closes #20230 from dongjoon-hyun/SPARK-23038.

(cherry picked from commit 7a3d0aa)
Signed-off-by: Felix Cheung <[email protected]>
asfgit pushed a commit that referenced this pull request Jan 14, 2018
@dongjoon-hyun
[SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
7022ef8 
## What changes were proposed in this pull request?

This PR aims to update the followings in `docker/spark-test`.

- JDK7 -> JDK8
Spark 2.2+ supports JDK8 only.

- Ubuntu 12.04.5 LTS(precise) -> Ubuntu 16.04.3 LTS(xeniel)
The end of life of `precise` was April 28, 2017.

## How was this patch tested?

Manual.

* Master
```
$ cd external/docker
$ ./build
$ export SPARK_HOME=...
$ docker run -v $SPARK_HOME:/opt/spark spark-test-master
CONTAINER_IP=172.17.0.3
...
18/01/11 06:50:25 INFO MasterWebUI: Bound MasterWebUI to 172.17.0.3, and started at http://172.17.0.3:8080
18/01/11 06:50:25 INFO Utils: Successfully started service on port 6066.
18/01/11 06:50:25 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
18/01/11 06:50:25 INFO Master: I have been elected leader! New state: ALIVE
```

* Slave
```
$ docker run -v $SPARK_HOME:/opt/spark spark-test-worker spark://172.17.0.3:7077
CONTAINER_IP=172.17.0.4
...
18/01/11 06:51:54 INFO Worker: Successfully registered with master spark://172.17.0.3:7077
```

After slave starts, master will show
```
18/01/11 06:51:54 INFO Master: Registering worker 172.17.0.4:8888 with 4 cores, 1024.0 MB RAM
```

Author: Dongjoon Hyun <[email protected]>

Closes #20230 from dongjoon-hyun/SPARK-23038.

(cherry picked from commit 7a3d0aa)
Signed-off-by: Felix Cheung <[email protected]>
@felixcheung
Copy link
Member

merged to master/2.3/2.2

@dongjoon-hyun
Copy link
Member Author

Thank you, @felixcheung and @srowen .

@dongjoon-hyun dongjoon-hyun deleted the SPARK-23038 branch January 14, 2018 07:56
MatthewRBruce pushed a commit to Shopify/spark that referenced this pull request Jul 31, 2018
@dongjoon-hyun @MatthewRBruce
[SPARK-23038][TEST] Update docker/spark-test (JDK/OS)
e9f5a0b 
## What changes were proposed in this pull request?

This PR aims to update the followings in `docker/spark-test`.

- JDK7 -> JDK8
Spark 2.2+ supports JDK8 only.

- Ubuntu 12.04.5 LTS(precise) -> Ubuntu 16.04.3 LTS(xeniel)
The end of life of `precise` was April 28, 2017.

## How was this patch tested?

Manual.

* Master
```
$ cd external/docker
$ ./build
$ export SPARK_HOME=...
$ docker run -v $SPARK_HOME:/opt/spark spark-test-master
CONTAINER_IP=172.17.0.3
...
18/01/11 06:50:25 INFO MasterWebUI: Bound MasterWebUI to 172.17.0.3, and started at http://172.17.0.3:8080
18/01/11 06:50:25 INFO Utils: Successfully started service on port 6066.
18/01/11 06:50:25 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
18/01/11 06:50:25 INFO Master: I have been elected leader! New state: ALIVE
```

* Slave
```
$ docker run -v $SPARK_HOME:/opt/spark spark-test-worker spark://172.17.0.3:7077
CONTAINER_IP=172.17.0.4
...
18/01/11 06:51:54 INFO Worker: Successfully registered with master spark://172.17.0.3:7077
```

After slave starts, master will show
```
18/01/11 06:51:54 INFO Master: Registering worker 172.17.0.4:8888 with 4 cores, 1024.0 MB RAM
```

Author: Dongjoon Hyun <[email protected]>

Closes apache#20230 from dongjoon-hyun/SPARK-23038.

(cherry picked from commit 7a3d0aa)
Signed-off-by: Felix Cheung <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@srowen srowen srowen left review comments

@felixcheung felixcheung felixcheung approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dongjoon-hyun @SparkQA @felixcheung @srowen