add new control plane classes

[cosmicBboy]

Signed-off-by: cosmicBboy [email protected]

TL;DR

Add Flyte* control plane objects to give programmatic access to flyte backend data structures.

Type

• Bug Fix
• Feature
• Plugin

Are all requirements met?

• Code completed
• Smoke tested
• Unit tests added
• Code documentation added
• Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

flyteorg/flyte#823

Follow-up issue

NA
OR
https://github.com/lyft/flyte/issues/

[codecov]

Codecov Report

Merging #425 (ef22e9e) into master (d77a773) will decrease coverage by 0.04%.
The diff coverage is 71.70%.

@@            Coverage Diff             @@
##           master     #425      +/-   ##
==========================================
- Coverage   84.39%   84.35%   -0.05%     
==========================================
  Files         344      357      +13     
  Lines       25835    26608     +773     
  Branches     2114     2205      +91     
==========================================
+ Hits        21804    22445     +641     
- Misses       3458     3564     +106     
- Partials      573      599      +26     

Impacted Files	Coverage Δ
flytekit/control_plane/tasks/executions.py	34.28% <34.28%> (ø)
flytekit/control_plane/component_nodes.py	56.45% <56.45%> (ø)
flytekit/control_plane/nodes.py	56.84% <56.84%> (ø)
flytekit/control_plane/workflow.py	66.66% <66.66%> (ø)
flytekit/control_plane/workflow_execution.py	70.58% <70.58%> (ø)
flytekit/control_plane/launch_plan.py	76.08% <76.08%> (ø)
flytekit/control_plane/interface.py	90.00% <90.00%> (ø)
flytekit/control_plane/tasks/task.py	93.61% <93.61%> (ø)
...lytekit/integration/control_plane/test_workflow.py	97.87% <97.87%> (ø)
flytekit/control_plane/identifier.py	100.00% <100.00%> (ø)
... and 28 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d77a773...ef22e9e. Read the comment docs.

[wild-endeavor]

can we remove this for now? this is old stuff.

[wild-endeavor]

is this function necessary if we're just reading from the template?

[wild-endeavor]

If you just do

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-411e700e2550> in <module>
----> 1 FlyteWorkflowExecution.fetch('flytesnacks', 'development', 'upu1b8xod3')

~/go/src/github.com/flyteorg/flytekit/flytekit/control_plane/workflow_execution.py in fetch(cls, project, domain, name)
    122     def fetch(cls, project: str, domain: str, name: str) -> "FlyteWorkflowExecution":
    123         return cls.promote_from_model(
--> 124             _flyte_engine.get_client().get_execution(
    125                 _core_identifier.WorkflowExecutionIdentifier(project=project, domain=domain, name=name)
    126             )

~/go/src/github.com/flyteorg/flytekit/flytekit/engines/flyte/engine.py in get_client()
     54 # This will be refactored away when we move to a heavier context object.
     55 def get_client() -> _SynchronousFlyteClient:
---> 56     return _FlyteClientManager(_platform_config.URL.get(), insecure=_platform_config.INSECURE.get()).client
     57
     58

~/go/src/github.com/flyteorg/flytekit/flytekit/engines/flyte/engine.py in __init__(self, *args, **kwargs)
     40         # TODO: use cases.
     41         if type(self)._CLIENT is None:
---> 42             c = _SynchronousFlyteClient(*args, **kwargs)
     43             type(self)._CLIENT = c
     44

~/go/src/github.com/flyteorg/flytekit/flytekit/clients/raw.py in __init__(self, url, insecure, credentials, options)
    151             self._channel = _insecure_channel(url, options=list((options or {}).items()))
    152         else:
--> 153             self._channel = _secure_channel(
    154                 url,
    155                 credentials or _ssl_channel_credentials(),

~/envs/flytekit/lib/python3.8/site-packages/grpc/__init__.py in secure_channel(target, credentials, options, compression)
   1968             "secure_channel cannot be called with insecure credentials." +
   1969             " Call insecure_channel instead.")
-> 1970     return _channel.Channel(target, () if options is None else options,
   1971                             credentials._credentials, compression)
   1972

~/envs/flytekit/lib/python3.8/site-packages/grpc/_channel.py in __init__(self, target, options, credentials, compression)
   1477         self._process_python_options(python_options)
   1478         self._channel = cygrpc.Channel(
-> 1479             _common.encode(target), _augment_options(core_options, compression),
   1480             credentials)
   1481         self._call_state = _ChannelCallState(self._channel)

~/envs/flytekit/lib/python3.8/site-packages/grpc/_common.py in encode(s)
     70         return s
     71     else:
---> 72         return s.encode('utf8')
     73
     74

AttributeError: 'NoneType' object has no attribute 'encode'

you just get a funny error. I ended up having to call:

FLYTE_PLATFORM_URL=localhost:30081 FLYTE_PLATFORM_INSECURE=True ipython

I think we need a better way to specify settings. We should probably start by respecting ~/.flyte/config if there's a file there.

[wild-endeavor]

Also, just the fact that I can get network setting related errors when calling fetch is making me lean towards the centralized idea we talked about.

[cosmicBboy]

you just get a funny error. I ended up having to call:

FLYTE_PLATFORM_URL=localhost:30081 FLYTE_PLATFORM_INSECURE=True ipython

I think we need a better way to specify settings. We should probably start by respecting ~/.flyte/config if there's a file there.

agreed, shall I make an issue for that?

also, let me know if this is good to merge