[fix][sec] Drop hdfs2 support, Upgrade hadoop3 to 3.4.0 and dnsjava to 3.6.2 to address CVE-2024-25638

[lhotari]

Motivation

In pulsar-io connectors, Hadoop 2 and Hadoop 3 versions before 3.4.0 depend on dnsjava 2.1.7 which is outdated and contains CVE-2024-25638 .

Upgrading dnsjava 2.1.7 to dnsjava 3.6.x isn't possible directly since dnsjava 3.x API isn't compatible with dnsjava 2.1.x API. dnsjava dependency has been excluded in this PR since hadoop 3.4.1 isn't released yet.

Modifications

• remove pulsar-io/hdfs2 connector since it depends on Hadoop 2 which is outdated and isn't maintained.
• upgrade hadoop3 version 3.4.0.
• upgrade hbase version used in pulsar-io/hbase to 2.6.0-hadoop3.
• use dependencyManagement to enforce the versions so that hadoop3 3.4.0 libraries are used consistently
  • hadoop3 doesn't have a bom so each depended dependency will have to be included in dependencyManagement
• exclude dnsjava dependencies since it's an optional dependency in hadoop
• enforce dnsjava to 3.6.2 to avoid any other dependency in pulling the vulnerable version.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 74.25%. Comparing base (bbc6224) to head (8de699c).
Report is 664 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #23411      +/-   ##
============================================
+ Coverage     73.57%   74.25%   +0.67%     
- Complexity    32624    34357    +1733     
============================================
  Files          1877     1940      +63     
  Lines        139502   146889    +7387     
  Branches      15299    16176     +877     
============================================
+ Hits         102638   109069    +6431     
- Misses        28908    29391     +483     
- Partials       7956     8429     +473     

Flag	Coverage Δ
inttests	27.40% <ø> (+2.81%)	⬆️
systests	24.38% <ø> (+0.06%)	⬆️
unittests	73.61% <ø> (+0.76%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 654 files with indirect coverage changes

[lhotari]

It looks like we need to wait for Hadoop 3.4.1 release to complete: https://issues.apache.org/jira/browse/HADOOP-19237

[lhotari]

"Exclude the META-INF/services/java.net.spi.InetAddressResolverProvider from your project" is mentioned as a workaround in dnsjava/dnsjava#338. Related Hadoop PR https://github.com/apache/hadoop/pull/7070/files . I guess that there might be ways to exclude and disable dnsjava too. However Hadoop 3.4.1 release would be useful.

[lhotari]

Let's wait for Hadoop 3.4.1 release that is in voting: https://lists.apache.org/thread/6dshj3nb26mlhzhmd8xkmw78l3zk896t