Introduce flyte Decks into flytekit

[pingsutw]

Signed-off-by: Kevin Su [email protected]

TL;DR

Allow user to visualize dataframe, preview markdown file on flyteconsole
Note: now It only works for local execution

To run an example:

1. pip install git+https://github.com/flyteorg/flytekit@deck && python example_deck.py
2. every task will output a HTML file like below contains input, output tab. In addition, Users can add custom tab if they want. For example, they can add a deck to see the markdown file or profile their panda dataframes.

output will be like

We should add some metadata in flyteidl to make flyteconsole be able to find those HTML files

message DeckMetadata {

  // Tab name.
  string name = 1;

  // HTML file.
 string path= 2;
}

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

• Add a renderer interface
• Add some default renderer such as FrameRenderer, MarkdownRenderer, FrameProfilingRenderer
• folks can easily add new renderer by extending renderer

Tracking Issue

flyteorg/flyte#2175

Follow-up issue

NA

[codecov]

Codecov Report

Merging #859 (948f00c) into master (1407dc7) will decrease coverage by 0.00%.
The diff coverage is 93.71%.

@@            Coverage Diff             @@
##           master     #859      +/-   ##
==========================================
- Coverage   86.47%   86.47%   -0.01%     
==========================================
  Files         234      239       +5     
  Lines       22875    23100     +225     
  Branches     2577     2604      +27     
==========================================
+ Hits        19782    19975     +193     
- Misses       2655     2680      +25     
- Partials      438      445       +7     

Impacted Files	Coverage Δ
flytekit/core/task.py	80.95% <ø> (ø)
flytekit/types/structured/bigquery.py	52.45% <0.00%> (ø)
flytekit/types/structured/structured_dataset.py	93.69% <76.92%> (-1.22%)	⬇️
flytekit/core/type_engine.py	86.03% <84.61%> (-0.03%)	⬇️
flytekit/deck/renderer.py	92.30% <92.30%> (ø)
flytekit/core/base_task.py	91.02% <95.00%> (+0.79%)	⬆️
flytekit/configuration/internal.py	95.65% <100.00%> (+0.19%)	⬆️
flytekit/core/context_manager.py	89.45% <100.00%> (+0.40%)	⬆️
flytekit/deck/__init__.py	100.00% <100.00%> (ø)
flytekit/deck/deck.py	100.00% <100.00%> (ø)
... and 18 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1407dc7...948f00c. Read the comment docs.

[wild-endeavor]

I know it's still wip but just at first glance...

• If we're going to have templating in the html, should we consider something like jinja2?
• Could you summarize the user-facing API somewhere.
• How will this work for dynamic tasks? Dynamic tasks are compiled at run-time - will these Deck objects interfere with compilation?
• How does this work with cached stuff?

@jsonporter could you take a look at the html page in the PR? just in case you have any thoughts.

[pingsutw]

Few updates,

• input and output tab to see task's input/output data
• Use jinja to dynamic create new html file, and each task will output one html file.
• html files are generated in run time, so it will not affect compilation.
• The task will not be executed when a cache hit occurs, so html files will not be generated.

[kumare3]

This is looking much better now

[kumare3]

Overall I love the PR, this is simple. But, I think we should support a simple html append also?

[kumare3]

cc @pradithya / @eapolinario

[kumare3]

cc @eapolinario we discussed 2 things right?

1. Move the feature to flytekit.experimental or add beta note to the api
2. on every @task(disable_deck=True) so that you can optionally disable the deck?

[pingsutw]

Few updates,

• Added disable_deck arg in @task
• Added beta note to the deck interface
• removed example_deck.py

[wild-endeavor]

nit

[wild-endeavor]

list of?