DX-67936 Netty 4.1.96 upgrade for CVE-2023-34462 io.netty:netty-handler 4.1.93.Final #36
Conversation
…ndler 4.1.93.Final
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or In the case of PARQUET issues on JIRA the title also supports: See also: |
|
https://netty.io/news/2023/07/27/4-1-96-Final.html We are happy to announce the release of netty 4.1.96.Final. This releases fixes a performance regression introduced in 4.1.95.Final and a small regression in HTTP/2. Because of that we urge everyone to upgrade as soon as possible. The most important changes are: Move the PoolThreadCache finalizer to a separate object (#13510) |
* apacheGH-36375: [Java] Added creating MapWriter in ComplexWriter. (apache#36351) (#32) Added new method rootAsMap() to ComplexWriter and implement it in ComplexWriterImpl for supporting map type. Previously in dremio side: When i trying to return map like output ComplexWrite with this code: org.apache.arrow.vector.complex.writer.BaseWriter.MapWriter mapWriter = out.rootAsList().map(false); mapWriter.startMap(); for (java.util.Map.Entry<java.lang.Integer, java.lang.Integer> element : map.entrySet()) { mapWriter.startEntry(); mapWriter.key().integer().writeInt((Integer) element.getKey()); mapWriter.value().integer().writeInt((Integer) element.getValue()); mapWriter.endEntry(); } mapWriter.endMap(); It use UnionMapWriter and generate schema like: EXPR$0: Map(false)<$data$: Union(Sparse, [1, 39])<struct: Struct<key: Int(32, true) not null, value: Int(32, true) not null> not null, map: Map(false)<entries: Struct<key: Int(32, true) not null, value: Int(32, true)> not null>>> But in OutputDerivation impl class where i should create output Complete type List<Field> children = Arrays.asList( CompleteType.INT.toField("key", false), CompleteType.INT.toField("value", false)); return new CompleteType(CompleteType.MAP.getType(), CompleteType.struct(children).toField(MapVector.DATA_VECTOR_NAME, false)); (This is only one valid case, because MapVector.initializeChildrenFromFields()) return EXPR$0::map<key::int32, value::int32> I found a place where it start using union - PromotableWriter.promoteToUnion. And in the end i have SCHEMA_CHANGE ERROR: Schema changed during projection. Schema was schema(EXPR$0::map<key::int32, value::int32>) but then changed to schema(EXPR$0::map<struct::struct<key::int32, value::int32>, map::map<key::int32, value::int32>>) * Closes: apache#36375 Authored-by: Ivan Chesnov <[email protected]> Signed-off-by: David Li <[email protected]> * DX-67936 Upgrade to Netty 4.1.96 for CVE-2023-34462 io.netty:netty-handler 4.1.93.Final (#36) * Update README_DREMIO for new commit. --------- Signed-off-by: David Li <[email protected]> Co-authored-by: lriggs <[email protected]> Co-authored-by: Logan Riggs <[email protected]>
I ran tpcds for this upgrade using arrow jars built from my personal fork. Looks good: https://jenkins.drem.io/job/AWS-EE-tpcds-perf-jdbc-client/1089/