Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add read support for original files of ACID/Transactional tables #2930

Closed
wants to merge 26 commits into from
Closed

Add read support for original files of ACID/Transactional tables #2930

Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
2aea6de
Changes to add read support for original files
Dec 13, 2019
01e644c
Refactoring acidinfo and addressing review comments
Jun 29, 2020
6446af3
product test fix
Jun 29, 2020
6e22ae6
Addressing review comments
Jul 5, 2020
30d56cc
Reverting logic specific to original files, don't read bucketId from …
Jul 6, 2020
e5c836c
PR comments fixes
Jul 21, 2020
d6e2a76
adding bucketing parameter to test
Jul 22, 2020
cd9c9e0
review fixes
findepi Jul 22, 2020
b7cc23e
reorder methods
findepi Jul 22, 2020
4732906
review fixes
findepi Jul 22, 2020
f829d83
review fixes
findepi Jul 22, 2020
e33f8c0
Remove createInternalHiveSplit and fix splittable
findepi Jul 22, 2020
284058d
Refac fileIterators usage
findepi Jul 22, 2020
e371387
Fix original files with filter
findepi Jul 22, 2020
e41d2e6
review fixes
findepi Jul 22, 2020
239591d
fixes to fixes
findepi Jul 22, 2020
ee1c88f
fixing product test and unit test
Jul 22, 2020
c0ead8f
addressing review comments
Jul 23, 2020
02d189e
dont read ACID bucket ID column for deltas
Jul 23, 2020
b6c0ff2
Fix .containsExactly -> .containsOnly and remove redundant ORDER BY
findepi Jul 23, 2020
c2086e7
Simplify per LO comment
findepi Jul 23, 2020
16245d5
Fix fmt
findepi Jul 23, 2020
ec99fb9
Add TODO comment
findepi Jul 23, 2020
2ddb40a
Remove bogus original files + bucketing v1/2 tests
findepi Jul 23, 2020
eb1c80c
empty commit for Ci
findepi Jul 23, 2020
5f688b0
empty commit for Ci
findepi Jul 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions presto-product-tests/src/main/java/io/prestosql/tests/hive/TestHiveTransactionalTable.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,22 +221,21 @@ public void testReadFullAcidWithOriginalFiles(boolean isPartitioned, BucketingTy
onHive().executeQuery("INSERT INTO TABLE " + tableName + hivePartitionString + " VALUES (22, 2)");
onHive().executeQuery("ALTER TABLE " + tableName + " SET " + hiveTableProperties(ACID, bucketingType));

String selectPrestoQuery = "SELECT col, fcol FROM " + tableName + " ORDER BY col";
// read with original files
assertThat(query(selectPrestoQuery)).containsOnly(row(21, 1), row(22, 2));
findepi marked this conversation as resolved.
Show resolved Hide resolved
assertThat(query("SELECT col, fcol FROM " + tableName)).containsOnly(row(21, 1), row(22, 2));
assertThat(query("SELECT col, fcol FROM " + tableName + " WHERE fcol = 1")).containsOnly(row(21, 1));

// read with original files and insert delta
onHive().executeQuery("INSERT INTO TABLE " + tableName + hivePartitionString + " VALUES (20, 3)");
assertThat(query(selectPrestoQuery)).containsOnly(row(20, 3), row(21, 1), row(22, 2));
assertThat(query("SELECT col, fcol FROM " + tableName)).containsOnly(row(20, 3), row(21, 1), row(22, 2));

// read with original files and delete delta
onHive().executeQuery("DELETE FROM " + tableName + " WHERE fcol = 2");
assertThat(query(selectPrestoQuery)).containsExactly(row(20, 3), row(21, 1));
assertThat(query("SELECT col, fcol FROM " + tableName)).containsOnly(row(20, 3), row(21, 1));

// read with original files and insert+delete delta (UPDATE)
onHive().executeQuery("UPDATE " + tableName + " SET col = 23 WHERE fcol = 1" + (isPartitioned ? " AND part_col = 2 " : ""));
assertThat(query(selectPrestoQuery)).containsExactly(row(20, 3), row(23, 1));
assertThat(query("SELECT col, fcol FROM " + tableName)).containsOnly(row(20, 3), row(23, 1));
}
finally {
onHive().executeQuery("DROP TABLE " + tableName);
Expand Down