Upgrading HDFS Repository Plugin to use HDFS 2.8.1 Client #25497
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WIP primary unit tests is passing, but it's leaking a thread. Gotta check if that's been there before Re-added protobuf fix. Fixed third party library checks. Unit tests fail because of leaked thread. Filter out terrible thread Just update to 2.8.1 while we're here Upgrading the HDFS fixture to 2.8.1 as well
jasontedor
approved these changes
Jun 30, 2017
plugins/repository-hdfs/build.gradle
Outdated
|
|
||
| // HDFS | ||
| 'io.netty.channel.ChannelOption', | ||
| 'io.netty.util.ReferenceCountUtil' |
There was a problem hiding this comment.
Can you sort these, for example, there are already io.netty excludes above?
| * @see "org.apache.hadoop.fs.FileSystem.Statistics.StatisticsDataReferenceCleaner" | ||
| * @see "org.apache.hadoop.fs.FileSystem.Statistics" | ||
| */ | ||
| public class HdfsClientThreadLeakFilter implements ThreadFilter { |
There was a problem hiding this comment.
Can it be package private and final?
| /** | ||
| * In Hadoop 2.8.0, there is a thread that is started by the filesystem to clean up old execution stats. | ||
| * This thread ignores all interrupts, catching InterruptedException, logging it, and continuing on | ||
| * with it's work. The thread is a daemon, so it thankfully does not stop the JVM from closing, and it |
rjernst
approved these changes
Jun 30, 2017
|
|
||
| @Override | ||
| public boolean reject(Thread t) { | ||
| return t.getName().equalsIgnoreCase(OFFENDING_THREAD_NAME); |
There was a problem hiding this comment.
Old habit of mine. I'll fix.
| @@ -32,7 +32,7 @@ esplugin { | |||
| apply plugin: 'elasticsearch.vagrantsupport' | |||
|
|
|||
| versions << [ | |||
| 'hadoop2': '2.7.1' | |||
| 'hadoop2': '2.8.1' | |||
There was a problem hiding this comment.
Did no dependency versions change? Just double checking; remember we don't use transitive dependencies, so upgrading means inspecting the new version's deps compared to what we currently pull in.
There was a problem hiding this comment.
Htrace is the only new dependency with a new version
| @@ -83,6 +85,8 @@ public static void main(String[] args) throws Exception { | |||
| cfg.set(DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY, keytabFile); | |||
| cfg.set(DFSConfigKeys.DFS_DATANODE_KEYTAB_FILE_KEY, keytabFile); | |||
| cfg.set(DFSConfigKeys.DFS_NAMENODE_ACLS_ENABLED_KEY, "true"); | |||
| // dfs.block.access.token.enable | |||
There was a problem hiding this comment.
nit: this comment line doesn't seem to add anything meaningful
jbaiera
added a commit
that referenced
this pull request
Jun 30, 2017
Hadoop 2.7.x libraries fail when running on JDK9 due to the version string changing to a single character. On Hadoop 2.8, this is no longer a problem, and it is unclear on whether the fix will be backported to the 2.7 branch. This commit upgrades our dependency of Hadoop for the HDFS Repository to 2.8.1.
jbaiera
added a commit
that referenced
this pull request
Jun 30, 2017
Hadoop 2.7.x libraries fail when running on JDK9 due to the version string changing to a single character. On Hadoop 2.8, this is no longer a problem, and it is unclear on whether the fix will be backported to the 2.7 branch. This commit upgrades our dependency of Hadoop for the HDFS Repository to 2.8.1.
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Jul 4, 2017
* master: (52 commits) Include shared/attributes.asciidoc from docs master Fixed page breaks for ICU Collation Keyword Fields Remove QueryParseContext (elastic#25486) [Test] Use a common testing class for all XContent filtering tests (elastic#25491) Tests fix - Significant terms/text aggs (elastic#25499) [DOCS] add docs for REST high level client index method (elastic#25501) Tests: Add Debian 9 (Stretch) to the packaging tests test: Run flush before upgrade and refresh after upgrade. Fix third party audit for repository-hdfs [TEST] Expect nodes getting disconnected quickly testPrimaryFailureIncreasesTerm should use assertBusy to wait for yellow Cleanup network / transport related settings (elastic#25489) Fix repository-hdfs plugin packaging test Remove allocation id from replica replication response (elastic#25488) Adjust BWC version on bad allocation request test Upgrading HDFS Repository Plugin to use HDFS 2.8.1 Client (elastic#25497) Adjust status on bad allocation explain requests Preliminary support for ARM Add doc note regarding explicit publish host Fix typo in name of test ...
clintongormley
added
:Distributed Coordination/Snapshot/Restore
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This should fix #25450. We are upgrading our version of the HDFS client used within the HDFS Repository Plugin in order to fix issues with Hadoop's client code parsing JDK9's version string. The HDFS Test fixture has also been updated to version 2.8.1.