[FLINK-33331] Update netty to 4.1.100 and arrow to 13.0.0 to make flink-python passing on java 21 #23664
Conversation
There was a problem hiding this comment.
@snuyanzin Thanks for updating the version. I have a minor question about the PR. PTAL, thanks.
| @@ -887,7 +887,7 @@ under the License. | |||
| <dependency> | |||
| <groupId>io.netty</groupId> | |||
| <artifactId>netty-bom</artifactId> | |||
| <version>4.1.70.Final</version> | |||
| <version>4.1.100.Final</version> | |||
There was a problem hiding this comment.
Hi, @snuyanzin Thank you for the netty version update. I have been working with netty recently and have noticed a hardcoded netty version in the flink-rpc-akka/pom.xml (
flink/flink-rpc/flink-rpc-akka/pom.xml
Line 98 in b759794
In addition, it appears that the original hardcoded version might not be accurate.
There was a problem hiding this comment.
Hi @TanYuxin-tyx
thanks for the question
the reason why upgrade of netty and arrow is required is failing of number of python tests while building with jdk21
I have my own pipepline setup with jdk21
example of error could be seen e.g. https://dev.azure.com/snuyanzin/flink/_build/results?buildId=2490&view=logs&j=fba17979-6d2e-591d-72f1-97cf42797c11&t=727942b6-6137-54f7-1ef9-e66e706ea068
same error was mentioned in FLINK-33331 description.
Since both netty and arrow require this removed constructor during python test they both need bumping the version.
By the way here it is the result of same pipeline with cherry-picked commit from this PR https://dev.azure.com/snuyanzin/flink/_build/results?buildId=2491&view=results
it could be seen that after that python tests with java 21 are passing (there are some other failures for other tests which are being fixed within other jira issue under FLINK-33163)
About the dependency you've mentioned: this dependency is used only for Pekko (former Akka) since current Pekko version can not run with netty 4.x... At the same time based on tests it is not an issue, looks like they don't use this removed constructor... and no need to bump the version
There was a problem hiding this comment.
One more note about flink-rpc-akka dependency
in theory it could be update in future within a separate task
however first there should be support of Netty 4 from Pekko's side which is done within apache/pekko#643
however it is not released yet...
There was a problem hiding this comment.
@snuyanzin OK, make sense. Thanks for your patient explanations.
|
Thanks for taking a look |
What is the purpose of the change
To support java 21.
Since JDK21 there was removed of constructor within https://bugs.openjdk.org/browse/JDK-8303083
Netty should support it and the support was added within netty/netty#13366 which is a part of 4.1.93.Final. Also need to bump arraw as well
Verifying this change
This change is a trivial rework
Does this pull request potentially affect one of the following parts:
@Public(Evolving): ( no)Documentation