[#3430] improvement(hdfs): Add a docker kerberized CI image to test kerberos authentication #3431
Conversation
dev/docker/hdfs/yarn-env.sh
Outdated
| @@ -0,0 +1,124 @@ | |||
| # Licensed to the Apache Software Foundation (ASF) under one or more | |||
There was a problem hiding this comment.
This is useless for yarn.
There was a problem hiding this comment.
So we can remove it freely?
|
Shall we have a kerberized hdfs only or add HMS also? |
As the version of HDFS in HMS and HDFS clusters is not the same( v2.6 & v3.1), I'm afraid that we cannot merge them and both are necessary. For the kerberized HMS, @qqqttt123 has prepared a draft PR for it. |
|
Why do we choose Hadoop 3.1 rather than Hadoop 2.6? |
yuqi1129
changed the title
The version of Hadoop in |
Hadoop 2.6 should also be worked. If we use Hadoop 2.6, then we can reuse the Hive image. I'm OK with either, you can choose which way is suitable for you. |
|
@qqqttt123 |
|
Another thing is that you still need to have a kerberized image for Hive, so this will be a duplicated work because you need to do it again when you test hive catalog with Keberos enabled. |
The question is: whether we need to provide a different version of HDFS such as Hadoop 3 for the Hadoop catalog, if this is unnecessary, I think @qqqttt123 should move forward #2364 and I can reused it. |
It's not necessary to have a separate image for Hadoop catalog. I've already used Hive image for IT, we can reuse it for kerberos enabled test.
I don't think @qqqttt123 has enough time to do so. It would be good if you can continue on his work. |
|
This is not needed, right? @yuqi1129 can you please close this PR? |
Yeah, I will close it. |
|
It's not necessary as #2368 has been merged. |
What changes were proposed in this pull request?
Add a docker kerberosized ci image.
Why are the changes needed?
We need this image to test kerberos authentication for HDFS.
Fix: #3430
Does this PR introduce any user-facing change?
N/A.
How was this patch tested?
Test locally, IT test in the next PR will confirm it.