DoltTable Plugin

[max-hoffman]

Signed-off-by: Max Hoffman [email protected]

TL;DR

Adds a DoltTable -- wrapper around pandas.DataFrame that commits changes to versioned database.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

hanging TODO's:

• code coverage
• metadata management?
• flytesnacks (separate PR)

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/lyft/flyte/issues/

Follow-up issue

NA
OR
https://github.com/lyft/flyte/issues/

[max-hoffman]

@wild-endeavor Something i was confused about before -- will CI run this script to install Dolt?

[wild-endeavor]

CI will run this https://github.com/flyteorg/flytekit/blob/master/.github/workflows/pythonbuild.yml#L37 so as long as it gets picked up from that we should be good.

I'm also thinking we'll need to move plugins into a separate repo sooner rather than later.

[wild-endeavor]

Can we rename the plugin from dolt to flytedolt or something? I need to rename sqlalchemy actually to "flytesqlalchemy". Under certain installations of pip dependencies, it thinks that the plugin is the real sqlalchemy instead, so it didn't pick up the real library.

[max-hoffman]

@wild-endeavor would something like this solve those problems?

I'm surprised the other plugins like papermill and pandera haven't had issues

[wild-endeavor]

is that common practice? using . in the name of the plugin?

[max-hoffman]

I think this would work too:

plugins/
    flytekitplugins.dolt/
        setup.py
        flytekitplugins/
            dolt/
                __init__.py

I'm not confident about how different namespacing formats impact pip clashes.

[wild-endeavor]

yeah i like the second approach. i think the top level one is the one messing things up.

[codecov]

Codecov Report

Merging #461 (fa298da) into master (8492223) will increase coverage by 0.07%.
The diff coverage is 97.33%.

@@            Coverage Diff             @@
##           master     #461      +/-   ##
==========================================
+ Coverage   84.36%   84.43%   +0.07%     
==========================================
  Files         357      360       +3     
  Lines       26630    26780     +150     
  Branches     2206     2211       +5     
==========================================
+ Hits        22467    22613     +146     
- Misses       3564     3566       +2     
- Partials      599      601       +2     

Impacted Files	Coverage Δ
...lytekitplugins.dolt/flytekitplugins/dolt/schema.py	92.98% <92.98%> (ø)
...tekitplugins.dolt/flytekitplugins/dolt/__init__.py	100.00% <100.00%> (ø)
plugins/tests/dolt/test_wf.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8492223...fa298da. Read the comment docs.

[wild-endeavor]

+1 but unf you have to fix some conflicts. I'll approve again after. just ping me.