Hive 3

[findepi]

Support Hive 3.1+ (HDP 3.1)

• Create product-test environments with Hive 3.1 Create product-test environments with Hive 3.1 #190
• Make product tests pass
  • Make product tests run on HDP 3 Enable HDP 3 product tests #1615
• Use HDP 3.1 defaults in HDP 3.1 product test environment Use HDP 3.1 defaults in HDP 3.1 product test environment #1841
• Run product tests on HDP 3 by default Use HDP 3.1 by default in product tests #1239
• Support reading Hive managed Transactional/ACID tables Support Hive managed Transactional/ACID tables #576
  • insert-only tables
  • ORC ACID (Read Hive ORC ACID tables #2068), (Support bucketed ACID tables #1591)
  • Support ORC ACID with base in raw format (Support ORC ACID with base in raw format #2292)
  • Support reading ACID/Transactional tables with "original files" (Support reading ACID/Transactional tables with "original files" #2293)
• Allow creation of and writing to Hive transactional/ACID tables Allow creation of and writing to Hive transactional/ACID tables in Presto #1956
  • Allow creation of transactional/ACID tables
  • Allow writes to transactional/ACID tables
• Accommodate to Hive timestamp changes https://issues.apache.org/jira/browse/HIVE-12192 "Hive should carry out timestamp computations in UTC" and related
  • Fix ORC reader (io.prestosql.plugin.hive.AbstractTestHive#testTypesOrc)
  • Fix Parquet reader (io.prestosql.plugin.hive.AbstractTestHive#testTypesParquet) (requires Hive 3.1.2, https://issues.apache.org/jira/browse/HIVE-21002, https://issues.apache.org/jira/browse/HIVE-21290)
    • proper defaults Set hive.parquet.time-zone to UTC by default #5139
  • Fix io.prestosql.plugin.hive.AbstractTestHive#testTypesRcBinary (requires https://issues.apache.org/jira/browse/HIVE-22167)
  • Add similar test for Avro (Fix handling of Avro timestamps created in Hive 3.1 #5144; may require Hive 3.1.2, https://issues.apache.org/jira/browse/HIVE-21002, https://issues.apache.org/jira/browse/HIVE-21291)
• TIMESTAMP behaviour does not match sql standard TIMESTAMP behaviour does not match sql standard #37 is related due to https://issues.apache.org/jira/browse/HIVE-12192 "Hive should carry out timestamp computations in UTC"
• Support Hive timestamp with local time zone type (Support Hive's TIMESTAMP WITH LOCAL TIME ZONE #1240)
• Hive bucket version 2 tables Add test for writing to Hive bucket version 2 tables #1201, Support Hive bucket version 2 tables #538
  • Hive bucket version 2 for timestamp Support Hive bucketing v2 for timestamp #1706
• Handle Hive's information_schema Hive views are not supported: 'information_schema.tables' for version >=315 #1192 Filter out Hive information_schema and sys #3008
• Handle Hive's sys schema Filter out Hive information_schema and sys #3008
• Handle Hive's new privileges Trino cannot query hive connector (hadoop/hive 3): Unsupported privilege name: ALTER #4253 (Add Hive ALTER privilege #4254, Add hive 3 missing privileges and principal #4307)
• Support Hive's DEFAULT Implement Column default constraint #2048
• Support Hive's NOT NULL Support NOT NULL constraint of Hive 3 tables #2923
• Support role as an owner of hive table Support role as an owner of hive table #5706
• Document capabilities & limitations Hive 3 support documentation #3013
• ...

Related:

• Hive 3.1: "Malformed ORC file. Invalid postscript length 17" when only one data-file in external table directory #456

[RugratsJ]

@findepi, I installed Presto 332 and HDP 3.1, when connecting to Hive and run select queries, I got Original non-ACID files in transactional tables are not supported. What should I do to at lease be able to query (read-only) to Hive 3? Thanks.

[findepi]

This is a high-level roadmap issue. Let's keep the discussion high level here only. @RugratsJ Please ping me on our slack.

[jiegzhan]

Added hive-3 missing privileges and principal:

#4254

#4307

[dain]

Is HDP end of life (https://endoflife.software/applications/big-data/hortonworks-data-platform-hdp)? Should we update the title to just Hive3?