support for external table in AWS

[gourav-sg]

Hi,
users building data lake in AWS, or in other systems, use EXTERNAL TABLES quite a lot. This gives their solutions high flexibility. For example having either a personal metastore or using Glue datacatalog.

It will be great if delta support as a storage class was included for HIVE metastores as well. I think we can and do read data using HIVE or SPARK directly from DELTA write locations, but it does give us duplicates, because its just reading the parquet files.

Regards,
Gourav

[tdas]

We are working towards improving our table support in OSS so that we can run all Delta table operations directly on table names defined in Hive metastore and other such catalogs. The challenge is to add the necessary customizations in Apache Spark to make DDLs work with Delta's setup.

[rtjarvis]

This is important for your use case and web be happy to contribute. How far have you got?

[pranavanand]

We are still working on this and are waiting for the Spark 3.0. This should ideally come out in the Delta Lake 0.7.0 release.

[gourav-sg]

HI, thanks a ton Pranav, its a great news. Thanks and Regards, Gourav Sengupta

…

On Thu, Mar 19, 2020 at 9:53 PM Pranav ***@***.***> wrote: We are still working on this and are waiting for the Spark 3.0. This should ideally come out in the Delta Lake 0.7.0 release. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJZLQ66LYZ4GGZQ5JCRTIDRIKH3XANCNFSM4HZPVBHQ> .

[tdas]

the table support is being tracked by this issue - #85

[kadprad]

I am trying to use delta lake using AWS DMS and Glue. Could you help me find relevant posts.

[gourav-sg]

Hi, I use EMR (never found any reason valid enough to use AWS glue ETL, ofcourse we use GLUE Datacatalog) and install delta in it as a part of bootstrap actions. Except for EMR 5.29 I think delta works fine in the rest. The latest release of Delta works fine using EMR 6.1.0. There is a chance that AWS team might suggest to use HUDI, but personally I have found that HUDI's API's are too confusing for me to work with, and even more confusing to change/ debug. Best of luck Regards, Gourav

…

On Sun, Nov 8, 2020 at 6:24 PM kadprad ***@***.***> wrote: I am trying to use delta lake using AWS DMS and Glue. Could you help me find relevant posts. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJZLQZCWORJEFUVVURPGCTSO3O4BANCNFSM4HZPVBHQ> .

[zsxwing]

This has been released in Delta 0.7.0 on Spark 3.0. See #85

[gourav-sg]

beautiful. Thanks a ton for this. I have been using some of those features with a smile. Regards, Gourav Sengupta

…

On Wed, Apr 7, 2021 at 4:36 AM Shixiong Zhu ***@***.***> wrote: This has been released in Delta 0.7.0 on Spark 3.0. See #85 <#85> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#73 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJZLQ3U4S7KKJ4A4B7EJ5TTHPHK5ANCNFSM4HZPVBHQ> .