Replace the use of ::arrow::field() with std::make_shared<arrow::Field>() for the creation of arrow::Field instances in ArrowSchema. #9875

JkSelf · 2024-05-21T03:22:19Z

JkSelf
May 21, 2024
Collaborator

When we enable static builds within gluten and run SQL for Parquet writing on CentOS 7, we encounter the following exception:

Upon investigation, it appears that the issue lies with velox's inability to directly call the ::arrow::field() method when creating a schema. We have found that modifying the code to create arrow::Field instances using the std::make_shared method resolves the issue in this commit JkSelf@b0eef85.

Does anyone know the reason for this problem?

JkSelf · 2024-05-21T03:22:58Z

JkSelf
May 21, 2024
Collaborator Author

@mbasmanova @majetideepak @Yuhta Do you have any input? Thanks.

6 replies

Yuhta May 22, 2024
Collaborator

No idea why it is not working. Whether we pass reference or not should not matter since it is copied into SchemaField anyway.

JkSelf May 23, 2024
Collaborator Author

@majetideepak @Yuhta I tried the following code. The issue still exists when calling the ::arrow::field() method.

auto field = ::arrow::field(node.name(), type, node.is_optional(), FieldIdMetadata(node.field_id()));
return PopulateLeaf(column_index, field, current_levels, ctx, parent, out);

majetideepak May 23, 2024
Collaborator

@JkSelf can you share the exact exception you are seeing? It is not clear from the description.

JkSelf May 27, 2024
Collaborator Author

@majetideepak @Yuhta The detailed exception is as follows. We identified that the exception occurs here when creating the arrow::Field.

*** Error in `/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9.x86_64/jre/bin/java': free(): invalid pointer: 0x00007f546fbff180 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81299)[0x7f564cd20299]
/tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so(+0x52031dc)[0x7f54681ed1dc]
/tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so(_ZN8facebook5velox7parquet5arrow5arrow14SchemaManifest4MakeEPKNS2_16SchemaDescriptorERKSt10shared_ptrIKN5arrow16KeyValueMetadataEERKNS2_21ArrowReaderPropertiesEPS4_+0x4b3)[0x7f54681ee853]
/tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so(_ZN8facebook5velox7parquet5arrow5arrow10FileWriter4MakeEPN5arrow10MemoryPoolESt10unique_ptrINS2_17ParquetFileWriterESt14default_deleteIS9_EESt10shared_ptrINS5_6SchemaEESD_INS2_21ArrowWriterPropertiesEEPS8_IS4_SA_IS4_EE+0x22e)[0x7f54681dcd0e]
/tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so(_ZN8facebook5velox7parquet5arrow5arrow10FileWriter4OpenERKN5arrow6SchemaEPNS5_10MemoryPoolESt10shared_ptrINS5_2io12OutputStreamEESB_INS2_16WriterPropertiesEESB_INS2_21ArrowWriterPropertiesEE+0x2d7)[0x7f54681dd577]
/tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so(_ZN8facebook5velox7parquet6Writer5flushEv+0x73a)[0x7f54681d230a]
/tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so(_ZN8facebook5velox7parquet6Writer5closeEv+0x1c)[0x7f54681d19cc]
/tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libgluten.so(Java_org_apache_gluten_datasource_DatasourceJniWrapper_close+0x1a5)[0x7f546e6677e5]
[0x7f5635018507]
======= Memory map: ========
80200000-153d00000 rw-p 00000000 00:00 0
153d00000-580100000 ---p 00000000 00:00 0
580100000-5c8f80000 rw-p 00000000 00:00 0
5c8f80000-800000000 ---p 00000000 00:00 0
800000000-801240000 rw-p 00000000 00:00 0
801240000-840000000 ---p 00000000 00:00 0
5565be4ef000-5565be4f0000 r-xp 00000000 08:32 1669624524                 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9.x86_64/jre/bin/java
5565be6ef000-5565be6f0000 r--p 00000000 08:32 1669624524                 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9.x86_64/jre/bin/java
5565be6f0000-5565be6f1000 rw-p 00001000 08:32 1669624524                 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.412.b08-1.el7_9.x86_64/jre/bin/java
5565bf699000-5565bf6ba000 rw-p 00000000 00:00 0                          [heap]
7f53eaa50000-7f53eaa51000 ---p 00000000 00:00 0
7f53eaa51000-7f53eb251000 rwxp 00000000 00:00 0
7f53eb251000-7f53eb2c3000 r--p 00000000 08:32 1880496206                 /tmp/jnilib-3423365617182979416.tmp
7f53eb2c3000-7f53eb36a000 r-xp 00072000 08:32 1880496206                 /tmp/jnilib-3423365617182979416.tmp
7f53eb36a000-7f53eb39c000 r--p 00119000 08:32 1880496206                 /tmp/jnilib-3423365617182979416.tmp
7f53eb39c000-7f53eb39d000 ---p 0014b000 08:32 1880496206                 /tmp/jnilib-3423365617182979416.tmp
7f53eb39d000-7f53eb3a5000 r--p 0014b000 08:32 1880496206                 /tmp/jnilib-3423365617182979416.tmp
7f53eb3a5000-7f53eb3a7000 rw-p 00153000 08:32 1880496206                 /tmp/jnilib-3423365617182979416.tmp
7f53eb3a7000-7f53eb4aa000 rw-p 00000000 00:00 0
7f53eb4aa000-7f53eb5aa000 ---p 00000000 00:00 0
7f53eb5aa000-7f53eb7aa000 rw-p 00000000 00:00 0
7f53eb7aa000-7f53eb9aa000 rw-p 00000000 00:00 0
7f53eb9aa000-7f53eb9ec000 r-xp 00000000 08:32 1880496212                 /tmp/snappy-1.1.8-f64f070d-d8ab-4d27-b3f1-343f177a1363-libsnappyjava.so
7f53eb9ec000-7f53ebbeb000 ---p 00042000 08:32 1880496212                 /tmp/snappy-1.1.8-f64f070d-d8ab-4d27-b3f1-343f177a1363-libsnappyjava.so
7f53ebbeb000-7f53ebbef000 rw-p 00041000 08:32 1880496212                 /tmp/snappy-1.1.8-f64f070d-d8ab-4d27-b3f1-343f177a1363-libsnappyjava.so
7f53ebbef000-7f53ebe00000 rw-p 00000000 00:00 0
7f53ebe00000-7f53ec000000 rw-p 00000000 00:00 0
7f53ec000000-7f53ec1e3000 rw-p 00000000 00:00 0
7f53ec1e3000-7f53f0000000 ---p 00000000 00:00 0
7f53f0000000-7f53f0a7e000 rw-p 00000000 00:00 0
7f53f0a7e000-7f53f4000000 ---p 00000000 00:00 0
7f53f4000000-7f53f4021000 rw-p 00000000 00:00 0
7f53f4021000-7f53f8000000 ---p 00000000 00:00 0
7f53f8000000-7f53f8b2b000 rw-p 00000000 00:00 0
7f53f8b2b000-7f53fc000000 ---p 00000000 00:00 0
7f53fc000000-7f53fc021000 rw-p 00000000 00:00 0
7f53fc021000-7f5400000000 ---p 00000000 00:00 0
7f5400000000-7f5400021000 rw-p 00000000 00:00 0
7f5400021000-7f5404000000 ---p 00000000 00:00 0
7f5404000000-7f5404021000 rw-p 00000000 00:00 0
7f5404021000-7f5408000000 ---p 00000000 00:00 0
7f5408000000-7f5408021000 rw-p 00000000 00:00 0
7f5408021000-7f540c000000 ---p 00000000 00:00 0
7f540c000000-7f540c021000 rw-p 00000000 00:00 0
7f540c021000-7f5410000000 ---p 00000000 00:00 0
7f5410000000-7f5410021000 rw-p 00000000 00:00 0
7f5410021000-7f5414000000 ---p 00000000 00:00 0
7f5414000000-7f5414e29000 rw-p 00000000 00:00 0
7f5414e29000-7f5418000000 ---p 00000000 00:00 0
7f5418000000-7f5418021000 rw-p 00000000 00:00 0
7f5418021000-7f541c000000 ---p 00000000 00:00 0
7f541c000000-7f541c021000 rw-p 00000000 00:00 0
7f541c021000-7f5420000000 ---p 00000000 00:00 0
7f5420000000-7f5420162000 rw-p 00000000 00:00 0
7f5420162000-7f5424000000 ---p 00000000 00:00 0
7f5424000000-7f5424022000 rw-p 00000000 00:00 0
7f5424022000-7f5428000000 ---p 00000000 00:00 0
7f5428000000-7f5428021000 rw-p 00000000 00:00 0
7f5428021000-7f542c000000 ---p 00000000 00:00 0
7f542c000000-7f542c0a6000 rw-p 00000000 00:00 0
7f542c0a6000-7f5430000000 ---p 00000000 00:00 0
7f5430000000-7f5430021000 rw-p 00000000 00:00 0
7f5430021000-7f5434000000 ---p 00000000 00:00 0
7f5434000000-7f5434021000 rw-p 00000000 00:00 0
7f5434021000-7f5438000000 ---p 00000000 00:00 0
7f5438000000-7f5438021000 rw-p 00000000 00:00 0
7f5438021000-7f543c000000 ---p 00000000 00:00 0
7f543c000000-7f543c021000 rw-p 00000000 00:00 0
7f543c021000-7f5440000000 ---p 00000000 00:00 0
7f5440000000-7f5440021000 rw-p 00000000 00:00 0
7f5440021000-7f5444000000 ---p 00000000 00:00 0
7f5444000000-7f5444021000 rw-p 00000000 00:00 0
7f5444021000-7f5448000000 ---p 00000000 00:00 0
7f5448000000-7f5448021000 rw-p 00000000 00:00 0
7f5448021000-7f544c000000 ---p 00000000 00:00 0
7f544c000000-7f544c021000 rw-p 00000000 00:00 0
7f544c021000-7f5450000000 ---p 00000000 00:00 0
7f5450000000-7f5450118000 rw-p 00000000 00:00 0
7f5450118000-7f5454000000 ---p 00000000 00:00 0
7f5454000000-7f5454021000 rw-p 00000000 00:00 0
7f5454021000-7f5458000000 ---p 00000000 00:00 0
7f5458000000-7f5458021000 rw-p 00000000 00:00 0
7f5458021000-7f545c000000 ---p 00000000 00:00 0
7f545c000000-7f545c021000 rw-p 00000000 00:00 0
7f545c021000-7f5460000000 ---p 00000000 00:00 0
7f54600c7000-7f54602c7000 rw-p 00000000 00:00 0
7f54602c7000-7f54604c7000 rw-p 00000000 00:00 0
7f54604c7000-7f54606c7000 rw-p 00000000 00:00 0
7f54606c7000-7f54608c7000 rw-p 00000000 00:00 0
7f54608c7000-7f5460ac7000 rw-p 00000000 00:00 0
7f5460ac7000-7f5460aca000 ---p 00000000 00:00 0
7f5460aca000-7f5460bc8000 rwxp 00000000 00:00 0
7f5460bc8000-7f5460bcb000 ---p 00000000 00:00 0
7f5460bcb000-7f5460cc9000 rwxp 00000000 00:00 0
7f5460cc9000-7f5460ccc000 ---p 00000000 00:00 0
7f5460ccc000-7f5460dca000 rwxp 00000000 00:00 0
7f5460dca000-7f5460dcd000 ---p 00000000 00:00 0
7f5460dcd000-7f5460ecb000 rwxp 00000000 00:00 0
7f5460ecb000-7f5460ece000 ---p 00000000 00:00 0
7f5460ece000-7f5460fcc000 rwxp 00000000 00:00 0
7f5460fcc000-7f54611cc000 rw-p 00000000 00:00 0
7f54611cc000-7f54613cc000 rw-p 00000000 00:00 0
7f54613cc000-7f54615cc000 rw-p 00000000 00:00 0
7f54615cc000-7f54617cc000 rw-p 00000000 00:00 0
7f54617cc000-7f54619cc000 rw-p 00000000 00:00 0
7f54619cc000-7f5461bcc000 rw-p 00000000 00:00 0
7f5461bcc000-7f5461dcc000 rw-p 00000000 00:00 0
7f5461dcc000-7f5461fcc000 rw-p 00000000 00:00 0
7f5461fcc000-7f5461fcf000 ---p 00000000 00:00 0
7f5461fcf000-7f54620cd000 rwxp 00000000 00:00 0
7f54620cd000-7f54620d0000 ---p 00000000 00:00 0
7f54620d0000-7f54621ce000 rwxp 00000000 00:00 0
7f54621ce000-7f54621d1000 ---p 00000000 00:00 0
7f54621d1000-7f54622cf000 rwxp 00000000 00:00 0
7f54622cf000-7f54622d2000 ---p 00000000 00:00 0
7f54622d2000-7f54623d0000 rwxp 00000000 00:00 0
7f54623d0000-7f54625d0000 rw-p 00000000 00:00 0
7f54625d0000-7f54627d0000 rw-p 00000000 00:00 0
7f54627d0000-7f54629d0000 rw-p 00000000 00:00 0
7f54629d0000-7f5462bd0000 rw-p 00000000 00:00 0
7f5462bd0000-7f5462dd0000 rw-p 00000000 00:00 0
7f5462dd0000-7f5462de6000 r-xp 00000000 08:32 1614198995                 /usr/lib64/libresolv-2.17.so
7f5462de6000-7f5462fe6000 ---p 00016000 08:32 1614198995                 /usr/lib64/libresolv-2.17.so
7f5462fe6000-7f5462fe7000 r--p 00016000 08:32 1614198995                 /usr/lib64/libresolv-2.17.so
7f5462fe7000-7f5462fe8000 rw-p 00017000 08:32 1614198995                 /usr/lib64/libresolv-2.17.so
7f5462fe8000-7f5462fea000 rw-p 00000000 00:00 0
7f5462fea000-7f5464a37000 r--p 00000000 08:32 572871422                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so
7f5464a37000-7f5469629000 r-xp 01a4d000 08:32 572871422                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so
7f5469629000-7f546a63c000 r--p 0663f000 08:32 572871422                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so
7f546a63c000-7f546a91d000 r--p 07651000 08:32 572871422                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so
7f546a91d000-7f546a975000 rw-p 07932000 08:32 572871422                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libvelox.so
7f546a975000-7f546aa66000 rw-p 00000000 00:00 0
7f546aafa000-7f546aafd000 ---p 00000000 00:00 0
7f546aafd000-7f546abfb000 rwxp 00000000 00:00 0
7f546abfb000-7f546adfb000 rw-p 00000000 00:00 0
7f546adfb000-7f546affb000 rw-p 00000000 00:00 0
7f546affb000-7f546b1fb000 rw-p 00000000 00:00 0
7f546b1fb000-7f546b3fb000 rw-p 00000000 00:00 0
7f546b3fb000-7f546b42a000 r-xp 00000000 08:32 1880408523                 /tmp/liblz4-java-7634425440583050089.so
7f546b42a000-7f546b629000 ---p 0002f000 08:32 1880408523                 /tmp/liblz4-java-7634425440583050089.so
7f546b629000-7f546b62a000 r--p 0002e000 08:32 1880408523                 /tmp/liblz4-java-7634425440583050089.so
7f546b62a000-7f546b62b000 rw-p 0002f000 08:32 1880408523                 /tmp/liblz4-java-7634425440583050089.so
7f546b62b000-7f546b82b000 rw-p 00000000 00:00 0
7f546b82b000-7f546ba2b000 rw-p 00000000 00:00 0
7f546ba2b000-7f546bc2b000 rw-p 00000000 00:00 0
7f546bc2b000-7f546be2b000 rw-p 00000000 00:00 0
7f546be2b000-7f546be2e000 ---p 00000000 00:00 0
7f546be2e000-7f546bf2c000 rwxp 00000000 00:00 0
7f546bf2c000-7f546bf2f000 ---p 00000000 00:00 0
7f546bf2f000-7f546c02d000 rwxp 00000000 00:00 0
7f546c02d000-7f546c030000 ---p 00000000 00:00 0
7f546c030000-7f546c12e000 rwxp 00000000 00:00 0
7f546c12e000-7f546c32e000 rw-p 00000000 00:00 0
7f546c32e000-7f546c52e000 rw-p 00000000 00:00 0
7f546c52e000-7f546c72e000 rw-p 00000000 00:00 0
7f546c72e000-7f546c92e000 rw-p 00000000 00:00 0
7f546c92e000-7f546c931000 ---p 00000000 00:00 0
7f546c931000-7f546ca2f000 rwxp 00000000 00:00 0
7f546ca2f000-7f546cc2f000 rw-p 00000000 00:00 0
7f546cc2f000-7f546cc32000 ---p 00000000 00:00 0
7f546cc32000-7f546cd30000 rwxp 00000000 00:00 0
7f546cd30000-7f546cf30000 rw-p 00000000 00:00 0
7f546cf30000-7f546cf33000 ---p 00000000 00:00 0
7f546cf33000-7f546d031000 rwxp 00000000 00:00 0
7f546d031000-7f546d034000 ---p 00000000 00:00 0
7f546d034000-7f546d132000 rwxp 00000000 00:00 0
7f546d132000-7f546d135000 ---p 00000000 00:00 0
7f546d135000-7f546d233000 rwxp 00000000 00:00 0
7f546d233000-7f546d236000 ---p 00000000 00:00 0
7f546d236000-7f546d334000 rwxp 00000000 00:00 0
7f546d334000-7f546d534000 rw-p 00000000 00:00 0
7f546d534000-7f546d734000 rw-p 00000000 00:00 0
7f546d734000-7f546d737000 ---p 00000000 00:00 0
7f546d737000-7f546d835000 rwxp 00000000 00:00 0
7f546d835000-7f546d838000 ---p 00000000 00:00 0
7f546d838000-7f546d936000 rwxp 00000000 00:00 0
7f546d936000-7f546d939000 ---p 00000000 00:00 0
7f546d939000-7f546da37000 rwxp 00000000 00:00 0
7f546da37000-7f546da3a000 ---p 00000000 00:00 0
7f546da3a000-7f546db38000 rwxp 00000000 00:00 0
7f546db38000-7f546dd38000 rw-p 00000000 00:00 0
7f546dd38000-7f546dd3b000 ---p 00000000 00:00 0
7f546dd3b000-7f546de39000 rwxp 00000000 00:00 0
7f546de39000-7f546de3c000 ---p 00000000 00:00 0
7f546de3c000-7f546df3a000 rwxp 00000000 00:00 0
7f546df3a000-7f546df3d000 ---p 00000000 00:00 0
7f546df3d000-7f546e03b000 rwxp 00000000 00:00 0
7f546e03b000-7f546e03e000 ---p 00000000 00:00 0
7f546e03e000-7f546e13c000 rwxp 00000000 00:00 0
7f546e13c000-7f546e33c000 rw-p 00000000 00:00 0
7f546e33c000-7f546e566000 r--p 00000000 08:32 572871421                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libgluten.so
7f546e566000-7f546e8be000 r-xp 0022a000 08:32 572871421                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libgluten.so
7f546e8be000-7f546e9be000 r--p 00582000 08:32 572871421                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libgluten.so
7f546e9be000-7f546e9dc000 r--p 00681000 08:32 572871421                  /tmp/gluten-caa98f9d-5f72-45c4-abad-f1ad43bc981d/jni/9f39df2f-fd6d-485e-ba21-2e389edb595c/gluten-3360085202245475481/libgluten.

Yuhta May 27, 2024
Collaborator

Looks like a JNI bug to me. Try a different JVM may gives different results.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace the use of ::arrow::field() with std::make_shared<arrow::Field>() for the creation of arrow::Field instances in ArrowSchema. #9875

{{title}}

Replies: 1 comment 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Replace the use of ::arrow::field() with std::make_shared<arrow::Field>() for the creation of arrow::Field instances in ArrowSchema. #9875

JkSelf May 21, 2024 Collaborator

Replies: 1 comment · 6 replies

JkSelf May 21, 2024 Collaborator Author

Yuhta May 22, 2024 Collaborator

JkSelf May 23, 2024 Collaborator Author

majetideepak May 23, 2024 Collaborator

JkSelf May 27, 2024 Collaborator Author

Yuhta May 27, 2024 Collaborator

JkSelf
May 21, 2024
Collaborator

Replies: 1 comment 6 replies

JkSelf
May 21, 2024
Collaborator Author

Yuhta May 22, 2024
Collaborator

JkSelf May 23, 2024
Collaborator Author

majetideepak May 23, 2024
Collaborator

JkSelf May 27, 2024
Collaborator Author

Yuhta May 27, 2024
Collaborator