Added ability to have unique table location for each iceberg table #6063

sshkvar · 2020-11-23T18:22:45Z

Added new iceberg configuration property iceberg.unique-table-location

By default this property = false, so table directory will have the same name as table.
In case iceberg.unique-table-location = true unique UUID will be added to the table directory name, so each table will have unique location

electrum

A few minor comments, otherwise looks good

presto-iceberg/src/main/java/io/prestosql/plugin/iceberg/IcebergConfig.java

presto-iceberg/src/main/java/io/prestosql/plugin/iceberg/IcebergMetadata.java

presto-iceberg/src/main/java/io/prestosql/plugin/iceberg/IcebergMetadataFactory.java

sshkvar · 2020-11-24T20:00:26Z

A few minor comments, otherwise looks good

@electrum Thanks for the comments, I have performed code fixes for all of them

sshkvar · 2020-11-26T14:59:44Z

@electrum is it possible to have this changes in the next release?

sshkvar · 2020-12-03T13:22:02Z

@electrum could you please approve this PR in case of all looks good?

sshkvar · 2020-12-07T15:18:14Z

@electrum I would be really appreciate for your review of this PR

raunaqmorarka

Please add test to AbstractTestIcebergSmoke to verify that some UUID as added to table location.

Does iceberg.unique-table-location have any purpose beyond making it possible for Trino to delete data on drop of table ?
If not, it would be easier to have just one flag iceberg.delete-files-on-table-drop and implement both unique table location and data removal behind that flag rather than requiring users to set 2 flags.

Should we also be providing this as a table property ? From user point of view it's easier to create a "managed" table using table property rather than changing config to set a catalog property.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergConfig.java

findepi

@sshkvar thanks for working on this.

Can you please rebase?

We should also have a test for this that would cover
CREATE + DROP
CREATE + RENAME + DROP

showing that table data gets correctly dropped.

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergConfig.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

findepi

Thanks for adding the test.
Overall LGTM. Some editorial comments in the test.

.../trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergTableWithCustomLocation.java

findepi · 2021-07-29T07:21:09Z

cc @rdblue

findepi · 2021-07-29T13:54:58Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/IcebergQueryRunner.java

+    public static DistributedQueryRunner createIcebergQueryRunner(Map<String, String> extraProperties,
+                                                                  FileFormat format,
+                                                                  List<TpchTable<?>> tables,
+                                                                  Optional<File> metastoreDirectory)


Please update formatting

all parameters on one line OR each parameter on its own line (including the first one)

for indentation please refer to https://github.com/airlift/codestyle/

Thanks for the link to code style.
Formatting updated

findepi · 2021-07-29T13:55:34Z

plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/IcebergQueryRunner.java

    public static DistributedQueryRunner createIcebergQueryRunner(
            Map<String, String> extraProperties,
            FileFormat format,
            List<TpchTable<?>> tables,
-            Optional<File> metastoreDirectory)
+            Optional<File> metastoreDirectory,
+            Optional<Map<String, String>> icebergPropertiesOverride)


We call it connectorProperties in other XxxQueryRunner classes.

also, no need for optional, since empty map has the same meaning

Map<String, String> connectorProperties)

also, please add it right after extraProperties (also seems usual)

findepi · 2021-07-29T13:58:40Z

.../trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergTableWithCustomLocation.java

+        File tempDir = Files.createTempDirectory("test_iceberg_v2").toFile();
+        metastoreDir = new File(tempDir, "iceberg_data");


Why two levels of nesting?
Would it be fine to have just

metastoreDir = Files.createTempDirectory("....");

?

however, it seems the test doesn't need to know the metastoreDir, so instead of setting it up (and cleaning it later), it should be sufficient to pass Optional.empty() for the metastoreDir, right?

we need metastoreDir because it is used in createTestingFileHiveMetastore(metastoreDir),
but I removed two levels nesting

findepi · 2021-07-29T13:59:04Z

.../trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergTableWithCustomLocation.java

+
+        return createIcebergQueryRunner(
+                ImmutableMap.of(),
+                ORC,


use default

new IcebergConfig().getFileFormat()

as a follow up we should remove format parameter and use the new connectorProperties instead

replaced with new IcebergConfig().getFileFormat()

findepi · 2021-07-29T14:00:57Z

.../trino-iceberg/src/test/java/io/trino/plugin/iceberg/TestIcebergTableWithCustomLocation.java

        assertTrue(table.isPresent(), "Table should exists");
        String location = table.get().getStorage().getLocation();
-        assertTrue(location.matches(format(".*%s-[0-9a-f]{32}", TABLE)), "Table location should have UUID suffix");
+        assertTrue(location.matches(format(".*%s-[0-9a-f]{32}", "table_with_uuid")), "Table location should have UUID suffix");


Do not use assertTrue with complex expressions. In case of failure, it produces unhelpful message.

Use

[org.assertj.core.api.Assertions.] assertThat(location).matches(....);

instead

findepi · 2021-08-03T10:41:35Z

@sshkvar could you please remove the merge commits from the git history?
you should be able to do this with git rebase.

sshkvar · 2021-08-03T11:10:09Z

@sshkvar could you please remove the merge commits from the git history?
you should be able to do this with git rebase.

done

findepi · 2021-08-03T11:40:08Z

@sshkvar thanks!
is the separation into 4 commits significant here? if not, can you squash them into single commit?
(again, git rebase may be useful here)

…table

sshkvar · 2021-08-03T12:13:00Z

@findepi I combined changes to single commit

findepi · 2021-08-03T13:36:02Z

Merged, thanks!

sshkvar · 2021-08-03T13:37:43Z

Merged, thanks!

Thanks you for the review!

cla-bot bot added the cla-signed label Nov 23, 2020

sshkvar requested a review from electrum November 24, 2020 09:18

electrum reviewed Nov 24, 2020

View reviewed changes

sshkvar mentioned this pull request Dec 8, 2020

Iceberg DROP table not removing data from s3 #5616

Closed

raunaqmorarka reviewed May 18, 2021

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergConfig.java Outdated Show resolved Hide resolved

findepi reviewed Jul 27, 2021

View reviewed changes

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergConfig.java Outdated Show resolved Hide resolved

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java Outdated Show resolved Hide resolved

findepi mentioned this pull request Jul 27, 2021

Spark: Added ability to add uuid suffix to the table location in Hive catalog apache/iceberg#2850

Closed

sshkvar force-pushed the issue-5632 branch from 6472980 to 42341ba Compare July 28, 2021 07:32

findepi reviewed Jul 29, 2021

View reviewed changes

findepi requested review from electrum and phd3 July 29, 2021 07:20

findepi reviewed Jul 29, 2021

View reviewed changes

findepi force-pushed the master branch from 8538e49 to 1f896ea Compare July 30, 2021 22:13

sshkvar force-pushed the issue-5632 branch from 4e7941a to 3599db5 Compare August 3, 2021 11:05

rebase: Added ability to have unique table location for each iceberg …

0a30f79

…table

sshkvar force-pushed the issue-5632 branch from 3599db5 to 0a30f79 Compare August 3, 2021 12:09

findepi approved these changes Aug 3, 2021

View reviewed changes

findepi merged commit bea74d5 into trinodb:master Aug 3, 2021

findepi added this to the 361 milestone Aug 3, 2021

findepi mentioned this pull request Aug 3, 2021

Release notes for 361 #8732

Closed

10 tasks

findinpath mentioned this pull request Feb 16, 2022

Delete Iceberg table data on DROP #11062

Merged

findepi mentioned this pull request May 24, 2022

Implement rename table support for Delta connector #11400

Closed

findepi mentioned this pull request Jun 24, 2022

Generate unique location when Delta table created with no explicit location #12980

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added ability to have unique table location for each iceberg table #6063

Added ability to have unique table location for each iceberg table #6063

sshkvar commented Nov 23, 2020

electrum left a comment

sshkvar commented Nov 24, 2020

sshkvar commented Nov 26, 2020

sshkvar commented Dec 3, 2020

sshkvar commented Dec 7, 2020

raunaqmorarka left a comment

findepi left a comment

findepi left a comment

findepi commented Jul 29, 2021

findepi Jul 29, 2021

sshkvar Jul 30, 2021

findepi Jul 29, 2021

sshkvar Jul 30, 2021

findepi Jul 29, 2021

findepi Jul 29, 2021

sshkvar Jul 30, 2021

findepi Jul 29, 2021

findepi Jul 29, 2021

sshkvar Jul 30, 2021

findepi Jul 29, 2021

sshkvar Jul 30, 2021

findepi commented Aug 3, 2021

sshkvar commented Aug 3, 2021

findepi commented Aug 3, 2021

sshkvar commented Aug 3, 2021

findepi commented Aug 3, 2021

sshkvar commented Aug 3, 2021

		File tempDir = Files.createTempDirectory("test_iceberg_v2").toFile();
		metastoreDir = new File(tempDir, "iceberg_data");

Added ability to have unique table location for each iceberg table #6063

Added ability to have unique table location for each iceberg table #6063

Conversation

sshkvar commented Nov 23, 2020

electrum left a comment

Choose a reason for hiding this comment

sshkvar commented Nov 24, 2020

sshkvar commented Nov 26, 2020

sshkvar commented Dec 3, 2020

sshkvar commented Dec 7, 2020

raunaqmorarka left a comment

Choose a reason for hiding this comment

findepi left a comment

Choose a reason for hiding this comment

findepi left a comment

Choose a reason for hiding this comment

findepi commented Jul 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findepi commented Aug 3, 2021

sshkvar commented Aug 3, 2021

findepi commented Aug 3, 2021

sshkvar commented Aug 3, 2021

findepi commented Aug 3, 2021

sshkvar commented Aug 3, 2021