Doc Hub proposal #303

pingsutw · 2022-06-23T10:41:17Z

Signed-off-by: Kevin Su [email protected]

TL;DR

https://docs.google.com/document/d/1wvCvM6pLQ17Zr0DcpS9es2vOFQsNeOaNN99ufGw8O1U/edit#heading=h.w5wnz0kbiild

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

Add new entity (descriptionEntity)
Add description entity to taskSpec and workflowSpec. These two specs are only used in the create request.
Add GetDescriptionEntity and ListDescriptionEntities endpoints

Tracking Issue

flyteorg/flyte#531

Follow-up issue

NA

Signed-off-by: Kevin Su <[email protected]>

codecov · 2022-06-27T17:40:13Z

Codecov Report

Merging #303 (8068970) into master (446b575) will not change coverage.
The diff coverage is n/a.

❗ Current head 8068970 differs from pull request most recent head fd49fac. Consider uploading reports for the commit fd49fac to get more accurate results

@@           Coverage Diff           @@
##           master     #303   +/-   ##
=======================================
  Coverage   75.46%   75.46%           
=======================================
  Files          18       18           
  Lines        1174     1174           
=======================================
  Hits          886      886           
  Misses        237      237           
  Partials       51       51

Flag	Coverage Δ
unittests	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
clients/go/admin/authtype_enumer.go	`27.27% <ø> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Kevin Su <[email protected]>

EngHabu

A great start! Should we also "enhance" the parameter descriptions in the same PR? allowing labels... etc.

protos/flyteidl/admin/task.proto

protos/flyteidl/admin/description_entity.proto

Signed-off-by: Kevin Su <[email protected]>

katrogan · 2022-09-09T22:08:19Z

hey @pingsutw is this proposal ready to be reviewed?

Signed-off-by: Kevin Su <[email protected]>

pingsutw · 2022-09-14T10:58:00Z

@katrogan yes, could you take a look, please?
Update:

Add tags to task/workflow/execution CreateRequest
Add descriptionEntity to task/workflow/launchPlan, so we can get entitySpec and descriptionEntity in one RPC call by using getRequest or ListRequest.

protos/flyteidl/admin/description_entity.proto

protos/flyteidl/admin/execution.proto

protos/flyteidl/admin/launch_plan.proto

protos/flyteidl/core/identifier.proto

Signed-off-by: Kevin Su <[email protected]>

katrogan · 2022-10-13T23:00:46Z

protos/flyteidl/admin/description_entity.proto

+// DescriptionEntity contains detailed description for the task/workflow.
+// Documentation could provide insight into the algorithms, business use case, etc.
+message DescriptionEntity {
+  // id represents the unique identifier of the description entity.


any reason not to add an identifier resource type for description entities?

katrogan · 2022-10-13T23:03:59Z

protos/flyteidl/admin/task.proto

@@ -51,6 +55,9 @@ message TaskList {
 message TaskSpec {
    // Template of the task that encapsulates all the metadata of the task.
    core.TaskTemplate template = 1;
+
+    // Represents the specification for Description.
+    DescriptionEntity description_entity        = 2;


Is this always going to be populated in requests to Get and List tasks?

TaskSpec is only used for TaskCreateRequest, and we use Task for get and list tasks

awesome! thanks for explaining

protos/flyteidl/admin/description_entity.proto

protos/flyteidl/admin/task.proto

protos/flyteidl/admin/workflow.proto

EngHabu · 2022-10-20T23:14:44Z

protos/flyteidl/service/admin.proto

+  // Fetch a :ref:`ref_flyteidl.admin.DescriptionEntity` object.
+  rpc GetDescriptionEntity (flyteidl.admin.ObjectGetRequest) returns (flyteidl.admin.DescriptionEntity) {
+    option (google.api.http) = {
+      get: "/api/v1/description_entities/{id.project}/{id.domain}/{id.name}/{id.version}"


do we need to put the resource type in the url as well?

Also, id refers to a NamedEntityIdentifier which doesn't have a version.. what is this supposed to point to?

yes, we should add resource_type.

cc @EngHabu one question. I got an error when adding {id.resource_type} to the GetDescriptionEntity endpoint.

$ make generate .... + mockery -dir=gen/pb-go/flyteidl/service/ -all -output=clients/go/admin/mocks Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType + mockery -dir=gen/pb-go/flyteidl/datacatalog/ -name=DataCatalogClient -output=clients/go/datacatalog/mocks Generating mock for: DataCatalogClient in file: clients/go/datacatalog/mocks/DataCatalogClient.go go generate ./... Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType Error parsing file: /Users/kevin/git/flyteidl/gen/pb-go/flyteidl/service/admin.pb.gw.go:2743:29: undeclared name: ResourceType

However, it works if I add {resource_type} to ListDescriptionEntities endpoint.

we use core.Identifier in ObjectGetRequest instead of NamedEntityIdentifier. there is a version in core.Identifier

Signed-off-by: Kevin Su <[email protected]>

generate_protos.sh

EngHabu · 2022-10-27T18:29:14Z

protos/flyteidl/core/identifier.proto

@@ -19,7 +19,7 @@ enum ResourceType {
 // Encapsulation of fields that uniquely identifies a Flyte resource.
 message Identifier {
    // Identifies the specific type of resource that this identifier corresponds to.
-    ResourceType resource_type  = 1;
+    core.ResourceType resource_type  = 1;


@evalsocket mind taking a look at this? do you have any ideas why this generates bad code?

Signed-off-by: Kevin Su <[email protected]>

* Add description entity Signed-off-by: Kevin Su <[email protected]> * Add id Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * few update Signed-off-by: Kevin Su <[email protected]> * update service Signed-off-by: Kevin Su <[email protected]> * update service Signed-off-by: Kevin Su <[email protected]> * Add description entity to task and workflow Signed-off-by: Kevin Su <[email protected]> * update des entity Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> * address comment Signed-off-by: Kevin Su <[email protected]> * update idl Signed-off-by: Kevin Su <[email protected]> * list description entity Signed-off-by: Kevin Su <[email protected]> * make generate Signed-off-by: Kevin Su <[email protected]> * make generate Signed-off-by: Kevin Su <[email protected]> * Update service name Signed-off-by: Kevin Su <[email protected]> * update endpoint Signed-off-by: Kevin Su <[email protected]> * update endpoint Signed-off-by: Kevin Su <[email protected]> * remove create_description_entity endpoint Signed-off-by: Kevin Su <[email protected]> * Add description to task/workflow Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * address comments Signed-off-by: Kevin Su <[email protected]> * address comments Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * Add id.resource_type Signed-off-by: Kevin Su <[email protected]> * undeclared name: ResourceType Signed-off-by: Kevin Su <[email protected]> * update wrong code manually Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Signed-off-by: Bernhard <[email protected]>

* Add description entity Signed-off-by: Kevin Su <[email protected]> * Add id Signed-off-by: Kevin Su <[email protected]> * wip Signed-off-by: Kevin Su <[email protected]> * few update Signed-off-by: Kevin Su <[email protected]> * update service Signed-off-by: Kevin Su <[email protected]> * update service Signed-off-by: Kevin Su <[email protected]> * Add description entity to task and workflow Signed-off-by: Kevin Su <[email protected]> * update des entity Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * typo Signed-off-by: Kevin Su <[email protected]> * address comment Signed-off-by: Kevin Su <[email protected]> * update idl Signed-off-by: Kevin Su <[email protected]> * list description entity Signed-off-by: Kevin Su <[email protected]> * make generate Signed-off-by: Kevin Su <[email protected]> * make generate Signed-off-by: Kevin Su <[email protected]> * Update service name Signed-off-by: Kevin Su <[email protected]> * update endpoint Signed-off-by: Kevin Su <[email protected]> * update endpoint Signed-off-by: Kevin Su <[email protected]> * remove create_description_entity endpoint Signed-off-by: Kevin Su <[email protected]> * Add description to task/workflow Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * address comments Signed-off-by: Kevin Su <[email protected]> * address comments Signed-off-by: Kevin Su <[email protected]> * fix tests Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * fix test Signed-off-by: Kevin Su <[email protected]> * Add id.resource_type Signed-off-by: Kevin Su <[email protected]> * undeclared name: ResourceType Signed-off-by: Kevin Su <[email protected]> * update wrong code manually Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> * Fixed tests Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Kevin Su <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]>

Add description entity

a56f648

Signed-off-by: Kevin Su <[email protected]>

pingsutw marked this pull request as draft June 24, 2022 19:29

Merged master

1f7af4b

Signed-off-by: Kevin Su <[email protected]>

pingsutw added 2 commits June 28, 2022 02:00

Add id

899ac99

Signed-off-by: Kevin Su <[email protected]>

wip

a642e84

Signed-off-by: Kevin Su <[email protected]>

EngHabu reviewed Jun 30, 2022

View reviewed changes

EngHabu assigned wild-endeavor Jun 30, 2022

pingsutw added 4 commits September 7, 2022 23:07

Merge branch 'master' of github.com:flyteorg/flyteidl into doc-hub

5013a89

few update

de901a7

Signed-off-by: Kevin Su <[email protected]>

update service

67e5e40

Signed-off-by: Kevin Su <[email protected]>

update service

fcfed61

Signed-off-by: Kevin Su <[email protected]>

pingsutw added 6 commits September 13, 2022 00:58

Add description entity to task and workflow

9938707

Signed-off-by: Kevin Su <[email protected]>

update des entity

d33ae81

Signed-off-by: Kevin Su <[email protected]>

update

4a5861d

Signed-off-by: Kevin Su <[email protected]>

Merge branch 'master' of github.com:flyteorg/flyteidl into doc-hub

ca3fc4f

nit

4a88f16

Signed-off-by: Kevin Su <[email protected]>

typo

7e9dfe5

Signed-off-by: Kevin Su <[email protected]>

katrogan reviewed Sep 14, 2022

View reviewed changes

pingsutw added 8 commits September 16, 2022 15:55

address comment

3bc09ce

Signed-off-by: Kevin Su <[email protected]>

update idl

1afd71a

Signed-off-by: Kevin Su <[email protected]>

list description entity

e8c3f2b

Signed-off-by: Kevin Su <[email protected]>

merged master

77f9920

Signed-off-by: Kevin Su <[email protected]>

make generate

5e66c76

Signed-off-by: Kevin Su <[email protected]>

make generate

3d884d3

Signed-off-by: Kevin Su <[email protected]>

Update service name

ef0a349

Signed-off-by: Kevin Su <[email protected]>

update endpoint

5b4d98c

Signed-off-by: Kevin Su <[email protected]>

katrogan reviewed Oct 13, 2022

View reviewed changes

wild-endeavor approved these changes Oct 14, 2022

View reviewed changes

Merge branch 'master' into doc-hub

47588e0

wild-endeavor previously approved these changes Oct 20, 2022

View reviewed changes

EngHabu reviewed Oct 21, 2022

View reviewed changes

address comments

972f9eb

Signed-off-by: Kevin Su <[email protected]>

pingsutw dismissed wild-endeavor’s stale review via 972f9eb October 21, 2022 19:37

pingsutw added 10 commits October 21, 2022 12:38

address comments

d23c279

Signed-off-by: Kevin Su <[email protected]>

fix tests

5014c97

Signed-off-by: Kevin Su <[email protected]>

nit

ea4901a

Signed-off-by: Kevin Su <[email protected]>

fix test

5685dcf

Signed-off-by: Kevin Su <[email protected]>

Add id.resource_type

300807b

Signed-off-by: Kevin Su <[email protected]>

undeclared name: ResourceType

767a476

Signed-off-by: Kevin Su <[email protected]>

update wrong code manually

20906d6

Signed-off-by: Kevin Su <[email protected]>

Merged master

02d5472

Signed-off-by: Kevin Su <[email protected]>

Fixed tests

8c964b2

Signed-off-by: Kevin Su <[email protected]>

Fixed tests

48fea31

Signed-off-by: Kevin Su <[email protected]>

wild-endeavor previously approved these changes Oct 25, 2022

View reviewed changes

Merged master

212159c

Signed-off-by: Kevin Su <[email protected]>

pingsutw dismissed wild-endeavor’s stale review via 212159c October 25, 2022 23:57

pingsutw commented Oct 26, 2022

View reviewed changes

generate_protos.sh Show resolved Hide resolved

EngHabu reviewed Oct 27, 2022

View reviewed changes

pingsutw added 2 commits October 31, 2022 13:29

Merged master

a109d5e

Signed-off-by: Kevin Su <[email protected]>

Merged master

fd49fac

Signed-off-by: Kevin Su <[email protected]>

wild-endeavor approved these changes Nov 5, 2022

View reviewed changes

EngHabu approved these changes Nov 5, 2022

View reviewed changes

pingsutw merged commit 6627432 into master Nov 7, 2022

pingsutw deleted the doc-hub branch November 7, 2022 18:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Doc Hub proposal #303

Doc Hub proposal #303

pingsutw commented Jun 23, 2022 •

edited

Loading

codecov bot commented Jun 27, 2022 •

edited

Loading

EngHabu left a comment

katrogan commented Sep 9, 2022

pingsutw commented Sep 14, 2022

katrogan Oct 13, 2022

katrogan Oct 13, 2022

pingsutw Oct 13, 2022 •

edited

Loading

katrogan Oct 13, 2022

EngHabu Oct 20, 2022

EngHabu Oct 21, 2022

pingsutw Oct 21, 2022 •

edited

Loading

pingsutw Oct 21, 2022

EngHabu Oct 27, 2022

Doc Hub proposal #303

Doc Hub proposal #303

Conversation

pingsutw commented Jun 23, 2022 • edited Loading

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

codecov bot commented Jun 27, 2022 • edited Loading

Codecov Report

EngHabu left a comment

Choose a reason for hiding this comment

katrogan commented Sep 9, 2022

pingsutw commented Sep 14, 2022

katrogan Oct 13, 2022

Choose a reason for hiding this comment

katrogan Oct 13, 2022

Choose a reason for hiding this comment

pingsutw Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

katrogan Oct 13, 2022

Choose a reason for hiding this comment

EngHabu Oct 20, 2022

Choose a reason for hiding this comment

EngHabu Oct 21, 2022

Choose a reason for hiding this comment

pingsutw Oct 21, 2022 • edited Loading

Choose a reason for hiding this comment

pingsutw Oct 21, 2022

Choose a reason for hiding this comment

EngHabu Oct 27, 2022

Choose a reason for hiding this comment

pingsutw commented Jun 23, 2022 •

edited

Loading

codecov bot commented Jun 27, 2022 •

edited

Loading

pingsutw Oct 13, 2022 •

edited

Loading

pingsutw Oct 21, 2022 •

edited

Loading