Add athena plugin #504
Add athena plugin #504
Conversation
Signed-off-by: Katrina Rogan <[email protected]>
|
|
||
| def get_custom(self, settings: SerializationSettings) -> Dict[str, Any]: | ||
| # This task is executed using the presto handler in the backend. | ||
| job = PrestoQuery(statement=self.query_template) |
There was a problem hiding this comment.
I stand corrected, I think it needs somethings set... they are just named differently... I think the minimum is the Database. We used Schema here to map it to...
There was a problem hiding this comment.
yep you're right! ty
Codecov Report
@@ Coverage Diff @@
## master #504 +/- ##
==========================================
+ Coverage 85.17% 85.19% +0.02%
==========================================
Files 366 369 +3
Lines 27766 27823 +57
Branches 2263 2266 +3
==========================================
+ Hits 23649 23705 +56
Misses 3496 3496
- Partials 621 622 +1
Continue to review full report at Codecov.
|
|
PTAL @kumare3 @EngHabu @wild-endeavor tested with flyteorg/flytesnacks#260 |
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
|
Let's change the top level folder name from athena to something else. Because this exists https://pypi.org/project/athena/ So |
Signed-off-by: Katrina Rogan <[email protected]>
|
@wild-endeavor thanks! done |
plugins/athena/setup.py
Outdated
|
|
||
| microlib_name = f"flytekitplugins-{PLUGIN_NAME}" | ||
|
|
||
| plugin_requires = ["flytekit>=0.18.0,<1.0.0"] |
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
|
@kumare3 did you still have concerns about this PR? maybe we should add a test for a dynamic task wrapping an athena task? |
|
|
||
| def get_custom(self, settings: SerializationSettings) -> Dict[str, Any]: | ||
| # This task is executed using the presto handler in the backend. | ||
| job = PrestoQuery(statement=self.query_template, schema=self.task_config.database) |
There was a problem hiding this comment.
Do you not want to fill/expose workgroup and catalog too?
Signed-off-by: Katrina Rogan <[email protected]>
Signed-off-by: Katrina Rogan <[email protected]>
|
|
||
| def get_custom(self, settings: SerializationSettings) -> Dict[str, Any]: | ||
| # This task is executed using the presto handler in the backend. | ||
| job = PrestoQuery( |
There was a problem hiding this comment.
My concern is these settings @wild-endeavor. These are not standard, but I guess ok
Signed-off-by: Haytham Abuelfutuh <[email protected]>
Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: Haytham Abuelfutuh <[email protected]> Signed-off-by: wild-endeavor <[email protected]>
Signed-off-by: Katrina Rogan [email protected]
TL;DR
Add athena plugin
Type
Are all requirements met?
Complete description
How did you fix the bug, make the feature etc. Link to any design docs etc
Tracking Issue
https://github.com/lyft/flyte/issues/
Follow-up issue
NA