Skip to content

Latest commit

 

History

History
34 lines (22 loc) · 1.97 KB

readme.md

File metadata and controls

34 lines (22 loc) · 1.97 KB

Updating the Thrift Hive Metastore client

Requirements

The thrift service mapping

Download the hive_metastore.thrift (and its required thrift files if any, place them in the same directory).

These two files are the mapping for the Thrift Service that the Hive Metastore is built on:

We are using the Hive Metastore version 3.0 (branch-3.0)

Thrift package

1 - First you need to install the thrift package so you can "compile" the thrift file into python code. You can refer to these tutorial for installing it: https://thrift-tutorial.readthedocs.io/en/latest/installation.html. We used the Thrift latest version (0.14.0)

2 - After installing thrift cli, you should open the directory where the hive_metastore.thrift (and others) is placed and run:

thrift --gen py hive_metastore.thrift
thrift --gen py fb303.thrift

3 - Now you have the new python code generated inside gen-py. Extract the classes and place them in the right directories.

4 - The generated files are huge, therefore be sure that the generated files directory names are ignored in the make commands style-check and apply-lint. So these files are not evaluated during the checks.

Manual adjustments to Thrift source files

Some auto-generated code from Thrift communication with Hive Metastore can be changed to provide more features, thus allowing a more customized integration.

The following PRs change the Thrift base files and/or source files. Check whether they need to be re-written in case of re-generation by the previous steps:

  • #45 - Changing type, from I16 to I32, of max_parts parameter from get_partitions method.