Use HDP 3.1 by default in product tests #1239
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Aug 2, 2019
Open
findepi
force-pushed
the
hdp3
branch
8 times, most recently
from
10f9eb2
to
c0bdcc3
Compare
findepi
force-pushed
the
hdp3
branch
11 times, most recently
from
229181f
to
cccc825
Compare
findepi
force-pushed
the
hdp3
branch
6 times, most recently
from
6576d1b
to
08b32a5
Compare
Despite https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-FloatingPointTypes saying "Floating point literals are assumed to be DOUBLE. Scientific notation is not yet supported." - Hive 1.2 supports scientific notation - hive 3 interprets decimal literals as decimals
`ALTER TABLE .. ENABLE OFFLINE` was removed in Hive 2.0
Hive 3's metastore returns basic statistics for a table even when we don't write them.
On HDP 3.1 new schema location is owned by `hive` on HDFS with `rwxr-xr-x` permissions. `alice` user cannot create a new table under the schema when HDFS impersonation is enabled.
Previously the `PRODUCT_TESTS_CONFIG=config-hdp3` configurations would still run on HDP 2.6.
HDP 3.1 gives better test coverage in the future. Tests on HDP 2.6 will be run in "branch builds".
The test expects all buckets are populated. However, Hive bucketing v1 uses float's bits as a hash, and for small float numbers, the last 5 bits (i.e. module 32 when there are 32 buckets) are usually zeros. We still got a few buckets (more than one) in `presto_test_bucketed_by_double_float`, because another hashed column was double.
There is a second `-Dhive.hadoop2.timeZone` on the same invocation.
Hive 3 is less permissive when it comes to schema evolution.
|
Obsolete. We run tests with Hive 3.1 and we currently don't run them on Hive 3.1 by default. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This gives better test coverage in the future.
Tests on HDP 2.6 will be run in "branch builds".
Related to #190, #37, #1218, #1188
Required for #1034