[#3751] feat(CI): Enable Apache Ranger plugin in Hive Docker image #3912

unknowntpo · 2024-06-19T22:57:59Z

What changes were proposed in this pull request?

Enable Apache Ranger plugin in Hive Docker image.
Add Ranger integration Hive permission test.
Both support Hive2 and Hive3
- To start the Hive container, you can to specify environment variables: HIVE_RUNTIME_VERSION:
- Hive version, currently support hive2(default) and hive3 DOCKER_ENV_RANGER_SERVER_URL:
- Ranger server URL DOCKER_ENV_RANGER_HIVE_REPOSITORY_NAME: Ranger Hive repository name
- DOCKER_ENV_RANGER_HDFS_REPOSITORY_NAME: Ranger HDFS repository name

Fix: #3751

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Local test.

TODO:

@xunliu needs to publish hive docker image
@unknowntpo needs to update docker-image-details.md

xunliu · 2024-06-29T02:09:52Z

hi @unknowntpo Thank you for your contributions.
Please help build two hive dockers to test Gravitino integration test.

HADOOP_VERSION=2.7.3 HIVE_VERSION=2.3.9 ./dev/docker/build-docker.sh --platform linux/arm64 --type hive --image datastrato/hive --tag 2.3.9-ranger
HADOOP_VERSION=3.1.0 HIVE_VERSION=3.1.3 ./dev/docker/build-docker.sh --platform linux/arm64 --type hive --image datastrato/hive --tag 3.1.3-ranger

unknowntpo · 2024-06-29T09:06:44Z

I've published docker images with these tags.

These two commands passed these tests:

CatalogHiveIT
CatalogIcebergRestIT
ProxyCatalogHiveIT

Note that if we run these tests with Intellij, we got not tests found for given include xxxx error,
in order to run tests, we needs to modify tag from:

@Tag("gravitino-docker-test")

to

@Tag("gravitino-docker-it")

dev/docker/hive/hive-dependency.sh

dev/docker/ranger/init-mysql.sql.template

dev/docker/ranger/start-ranger-services.sh

xunliu · 2024-06-29T14:26:02Z

I've published docker images with these tags.

These two commands passed these tests:

CatalogHiveIT

CatalogIcebergRestIT

ProxyCatalogHiveIT

Note that if we run these tests with Intellij, we got not tests found for given include xxxx error, in order to run tests, we needs to modify tag from:
@Tag("gravitino-docker-test")
to
@Tag("gravitino-docker-it")

hi @unknowntpo
Currently Gravitino didn't automatic run Docker integration test.
If you need run Docker IT in your local, please set skipDockerTests = false in
the gradle.properties:L29

So, we didn't need to modify @Tag("gravitino-docker-test").

dev/docker/hive/core-site.xml

dev/docker/hive/Dockerfile

xunliu · 2024-07-04T09:27:50Z

@mchades @yuqi1129 @jerryshao Please help me review this PR, Thanks.

...c/test/java/com/datastrato/gravitino/integration/test/authorization/ranger/RangerHiveIT.java

dev/docker/hive/Dockerfile

integration-test/build.gradle.kts

...c/test/java/com/datastrato/gravitino/integration/test/authorization/ranger/RangerHiveIT.java

...t/src/test/java/com/datastrato/gravitino/integration/test/authorization/ranger/RangerIT.java

.../src/test/java/com/datastrato/gravitino/integration/test/authorization/ranger/RangerRef.java

yuqi1129 · 2024-07-05T03:30:03Z

@jerryshao @mchades , Please help to review it, I'm not quite familiar with Ranger and can't help too much with Ranger-related code.

.../src/test/java/com/datastrato/gravitino/catalog/hadoop/integration/test/HadoopCatalogIT.java

mchades

please check the license of all new files

dev/docker/hive/Dockerfile

.../test/java/com/datastrato/gravitino/integration/test/authorization/ranger/RangerDefines.java

yuqi1129 · 2024-07-08T14:30:54Z

@unknowntpo @xunliu Please resolve the comments and conflicts.

unknowntpo · 2024-07-18T05:08:42Z

I've published new hive image to my docker hub repo for testing, to use this image, please set this line to integration-test/build.gradle.kts:

      environment("GRAVITINO_CI_RANGER_HIVE_DOCKER_IMAGE", "unknowntpo/gravitino-ci-hive:ranger-plugin") // Support Ranger plugin

Co-authored-by: xunliu <[email protected]> Co-authored-by: Eric Chang <[email protected]>

… package

dev/docker/build-docker.sh

dev/docker/hive/Dockerfile

docs/publish-docker-images.md

dev/docker/build-docker.sh

docs/publish-docker-images.md

xunliu

@unknowntpo Thank you for your contributions.
LGTM

docs/publish-docker-images.md

dev/docker/build-docker.sh

yuqi1129

LGTM

unknowntpo force-pushed the issue-3751 branch from eeaaea3 to 4a5c90f Compare June 19, 2024 23:06

unknowntpo force-pushed the issue-3751 branch 5 times, most recently from 3655168 to a3947c2 Compare June 28, 2024 13:54

unknowntpo marked this pull request as ready for review June 28, 2024 13:55

unknowntpo force-pushed the issue-3751 branch from 5985999 to 06eb84b Compare June 28, 2024 13:57

unknowntpo changed the title ~~WIP: Issue 3751~~ Enable Apache Ranger plugin in Hive Docker image Jun 28, 2024

unknowntpo force-pushed the issue-3751 branch from 060090e to 0be5292 Compare June 29, 2024 09:11

xunliu reviewed Jun 29, 2024

View reviewed changes

dev/docker/hive/hive-dependency.sh Outdated Show resolved Hide resolved

dev/docker/ranger/init-mysql.sql.template Outdated Show resolved Hide resolved

dev/docker/ranger/start-ranger-services.sh Outdated Show resolved Hide resolved

xunliu reviewed Jun 29, 2024

View reviewed changes

dev/docker/hive/core-site.xml Outdated Show resolved Hide resolved

xunliu changed the title ~~Enable Apache Ranger plugin in Hive Docker image~~ [#3751] feat(CI): Enable Apache Ranger plugin in Hive Docker image Jun 29, 2024

xunliu reviewed Jun 29, 2024

View reviewed changes

dev/docker/hive/Dockerfile Outdated Show resolved Hide resolved

dev/docker/hive/Dockerfile Outdated Show resolved Hide resolved

xunliu force-pushed the issue-3751 branch from d134efb to 6e9d239 Compare July 3, 2024 05:41

xunliu requested review from jerryshao, yuqi1129 and mchades July 4, 2024 09:29

yuqi1129 reviewed Jul 4, 2024

View reviewed changes

xunliu force-pushed the issue-3751 branch from 81159cd to 38ef88d Compare July 4, 2024 13:43

yuqi1129 reviewed Jul 5, 2024

View reviewed changes

.../src/test/java/com/datastrato/gravitino/catalog/hadoop/integration/test/HadoopCatalogIT.java Outdated Show resolved Hide resolved

mchades reviewed Jul 5, 2024

View reviewed changes

dev/docker/hive/Dockerfile Show resolved Hide resolved

.../test/java/com/datastrato/gravitino/integration/test/authorization/ranger/RangerDefines.java Outdated Show resolved Hide resolved

xunliu force-pushed the issue-3751 branch 2 times, most recently from b18d895 to e6a45a3 Compare July 11, 2024 02:25

mchades assigned unknowntpo Jul 17, 2024

unknowntpo force-pushed the issue-3751 branch from d3349a6 to 19021a0 Compare July 17, 2024 14:25

xunliu and others added 10 commits July 18, 2024 21:09

feat(ranger-plugin-hive): add docker config

3b73cb8

Co-authored-by: xunliu <[email protected]> Co-authored-by: Eric Chang <[email protected]>

fix(docker/hive): add missing build-arg

28666fe

fix(docker/hive): use datastrato/apache-ranger released ranger plugin…

beab570

… package

revert unnecessary change in ranger docker image

0cd4672

add properties to core-site.xml for hive

a031143

fix(docker/hive): fix wrong package name of hadoop

2cd963a

fix(docker/hive): fix wrong package name of hive

b586b22

Add ranger hive IT

beb1d68

Fixed Typo

7cb0b0f

WIP: support hive2,hive3,hadoop2,hadoop3 in one image

f743ee4

xunliu force-pushed the issue-3751 branch 5 times, most recently from e2dc83b to 5c0f5f5 Compare July 19, 2024 01:39

Improve hive2 and hive3

012433f

xunliu force-pushed the issue-3751 branch from 5c0f5f5 to 012433f Compare July 19, 2024 01:45

xunliu reviewed Jul 19, 2024

View reviewed changes

dev/docker/build-docker.sh Outdated Show resolved Hide resolved

dev/docker/hive/Dockerfile Outdated Show resolved Hide resolved

docs/publish-docker-images.md Outdated Show resolved Hide resolved

unknowntpo added 4 commits July 19, 2024 12:24

refactor(hive/Dockerfile): use ${} to wrap ZK_HOME variable

77c20b2

refactor(publish-docker-images.md): rollback changes

8449bff

refactor(dev/docker/build-docker.sh): rollback changes

96a419b

Merge branch 'main' into issue-3751

1855a1b

xunliu reviewed Jul 19, 2024

View reviewed changes

dev/docker/build-docker.sh Outdated Show resolved Hide resolved

docs/publish-docker-images.md Outdated Show resolved Hide resolved

fix

91f4842

xunliu approved these changes Jul 19, 2024

View reviewed changes

docs/publish-docker-images.md Outdated Show resolved Hide resolved

dev/docker/build-docker.sh Outdated Show resolved Hide resolved

yuqi1129 approved these changes Jul 19, 2024

View reviewed changes

yuqi1129 merged commit 3c4d906 into apache:main Jul 19, 2024
33 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#3751] feat(CI): Enable Apache Ranger plugin in Hive Docker image #3912

[#3751] feat(CI): Enable Apache Ranger plugin in Hive Docker image #3912

unknowntpo commented Jun 19, 2024 •

edited by xunliu

Loading

xunliu commented Jun 29, 2024

unknowntpo commented Jun 29, 2024

xunliu commented Jun 29, 2024

xunliu commented Jul 4, 2024

yuqi1129 commented Jul 5, 2024

mchades left a comment

yuqi1129 commented Jul 8, 2024

unknowntpo commented Jul 18, 2024 •

edited

Loading

xunliu left a comment

yuqi1129 left a comment

[#3751] feat(CI): Enable Apache Ranger plugin in Hive Docker image #3912

[#3751] feat(CI): Enable Apache Ranger plugin in Hive Docker image #3912

Conversation

unknowntpo commented Jun 19, 2024 • edited by xunliu Loading

What changes were proposed in this pull request?

Does this PR introduce any user-facing change?

How was this patch tested?

xunliu commented Jun 29, 2024

unknowntpo commented Jun 29, 2024

xunliu commented Jun 29, 2024

xunliu commented Jul 4, 2024

yuqi1129 commented Jul 5, 2024

mchades left a comment

Choose a reason for hiding this comment

yuqi1129 commented Jul 8, 2024

unknowntpo commented Jul 18, 2024 • edited Loading

xunliu left a comment

Choose a reason for hiding this comment

yuqi1129 left a comment

Choose a reason for hiding this comment

unknowntpo commented Jun 19, 2024 •

edited by xunliu

Loading

unknowntpo commented Jul 18, 2024 •

edited

Loading