Visualize graphs on command line #75

kumare3 · 2021-05-28T00:17:36Z

Signed-off-by: Ketan Umare [email protected]

TL;DR

Uses graphviz to visualize flyte workflows on commandline. Example of branch node

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Complete description

Uses graphviz to visualize flyte workflows on commandline

jsonporter · 2021-06-01T23:55:15Z

Graph looks good (once tests pass)

katrogan · 2021-06-02T00:19:58Z

pkg/visualize/graphviz.go

+	}
+	if n.Metadata != nil && n.Metadata.Name != "" {
+		v := strings.LastIndexAny(n.Metadata.Name, ".")
+		gn.SetLabel(fmt.Sprintf("%s [%s]", n.Metadata.Name[v+1:], t.Template.Type))


nit: in the examples it looks like the task type always goes outside the bounding box for the task node - is this fixable here?

hmm, fixable probably by using a better way of rendering

what if we do the beautification as a separate pass?

💯 sounds good!

yindia · 2021-06-03T15:15:33Z

@pmahindrakar-oss @kumare3 @@EngHabu

This pr need CGO enable in gobuild that's why it's failing. Goreleaser doesn't support CGO build. I explored some solution around this problem. Community is building CGO enabled build in container, They use different images for different PLATFORM/ARC and in our case it become complicated because we are generating multiple binary and may be we will add new binaries in future
Docker images: https://github.com/elastic/golang-crossbuild

second problem is that we use goreleaser for generating install.sh that we use in our docker images for installing flytectl, That will break

Third problem is brew, Currently brew package is published by goreleaser or else we need to do this manually

Need your thoughts on this. Alternative option is to use different library for graph (https://github.com/awalterschulze/gographviz)

Issue : #1082

kumare3 · 2021-06-03T15:54:46Z

@evalsocket this is really sad. The reason I used this library is because it brings the c-bindings with it. I.e no need of external dot library. Can we limit some go packages only for some platforms like osx

yindia · 2021-06-03T15:57:01Z

@kumare3 not sure. This lib look strong https://github.com/awalterschulze/gographviz. I discussed with @pmahindrakar-oss and he is exploring the capabilities of this lib.

kumare3 · 2021-06-03T15:58:14Z

The lib will only print a dot graph - and users will have to use an online renderer

kumare3 · 2021-06-03T15:58:24Z

I guess maybe we have to do this

yindia · 2021-06-03T17:07:09Z

ok

yindia · 2021-06-05T16:43:40Z

@kumare3 @pmahindrakar-oss checkout influxdb release process. They are using goreleaser for releasing binary with CGO enabled https://github.com/influxdata/influxdb/blob/master/.goreleaser.yml

kumare3 · 2021-06-06T04:31:18Z

@kumare3 @pmahindrakar-oss checkout influxdb release process. They are using goreleaser for releasing binary with CGO enabled https://github.com/influxdata/influxdb/blob/master/.goreleaser.yml

@evalsocket this is interesting? Why can't we do this - seems simple

kumare3 · 2021-06-06T04:32:57Z

We can try in this or

pmahindrakar-oss · 2021-06-06T05:43:36Z

@evalsocket let us know how it goes with the CGO changes similar to influxdb which you have suggested.

The changes with other library are also almost done aswell which dumps the dot string which can be used in online or commandline renderer for dot files. I will wait for your results to see if we want to change to complete go library for graphviz or use the existing one.

I used this for eg :

https://tinyurl.com/h6m7559j
Note : hyphens in names have been replaced by underscores since graphviz doesn't like those characters. (https://stackoverflow.com/questions/14958346/dot-dash-in-name)

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

* Using go graphviz library to print dot file Signed-off-by: Prafulla Mahindrakar <[email protected]> * Rebased Signed-off-by: Prafulla Mahindrakar <[email protected]>

Signed-off-by: Prafulla Mahindrakar <[email protected]>

* Linter fixes Signed-off-by: Prafulla Mahindrakar <[email protected]> * Added some coverage Signed-off-by: Prafulla Mahindrakar <[email protected]> * Added interface for the mehtods used from graphviz Signed-off-by: Prafulla Mahindrakar <[email protected]> * Fixed the workflow config flag file and bug fixes Signed-off-by: Prafulla Mahindrakar <[email protected]> (cherry picked from commit 77be561) * Added unit tests Signed-off-by: Prafulla Mahindrakar <[email protected]>

Signed-off-by: Prafulla Mahindrakar <[email protected]>

kumare3 requested a review from pmahindrakar-oss May 28, 2021 00:17

kumare3 requested review from EngHabu and wild-endeavor as code owners May 28, 2021 00:17

pmahindrakar-oss force-pushed the visualization branch from c82472e to 3e46732 Compare May 30, 2021 12:09

kumare3 changed the title ~~[wip]: Visualize graphs on command line~~ Visualize graphs on command line Jun 1, 2021

katrogan reviewed Jun 2, 2021

View reviewed changes

kumare3 requested a review from yindia June 2, 2021 05:07

yindia force-pushed the visualization branch 7 times, most recently from 1846c8e to 03cc7cf Compare June 8, 2021 10:06

kumare3 mentioned this pull request Jun 8, 2021

Correctly render static sub-workflow nodes flyteorg/flyte#357

Closed

yindia force-pushed the visualization branch 4 times, most recently from 31fefdf to 5b530ac Compare June 9, 2021 18:52

yindia force-pushed the visualization branch 5 times, most recently from a1440be to 3a929a6 Compare June 10, 2021 13:05

pmahindrakar-oss force-pushed the visualization branch 2 times, most recently from d938e10 to 990713a Compare June 17, 2021 08:41

kumare3 force-pushed the visualization branch 2 times, most recently from 3414179 to 990713a Compare June 17, 2021 23:01

pmahindrakar-oss force-pushed the visualization branch from 3528d2f to 449576b Compare June 18, 2021 08:12

kumare3 and others added 13 commits June 18, 2021 14:48

[wip]: Visualize graphs on command line

4ab2b4e

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

[wip] working version

6b57157

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

updated

cc9c35a

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

Working nested graphs - v0

b8908c1

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

better graph

6f05a5e

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

update todos

ad4d4c3

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

improved graph

644ff2b

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

Working with subworkflows

d420faf

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

subworkflows rendering

7dbb47b

Signed-off-by: Ketan Umare <[email protected]> Signed-off-by: Prafulla Mahindrakar <[email protected]>

Using go graphviz library to print dot file (#97)

517f539

* Using go graphviz library to print dot file Signed-off-by: Prafulla Mahindrakar <[email protected]> * Rebased Signed-off-by: Prafulla Mahindrakar <[email protected]>

Added output option for dot and doturl (#101)

b347c0c

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Added more coverage

f4711af

Signed-off-by: Prafulla Mahindrakar <[email protected]>

pmahindrakar-oss force-pushed the visualization branch from 449576b to f4711af Compare June 18, 2021 10:07

pmahindrakar-oss added 3 commits June 18, 2021 18:20

Fixed the doc

731bcc6

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Generated the docs

f85d6b6

Signed-off-by: Prafulla Mahindrakar <[email protected]>

Fixed the docs

9d1c85a

Signed-off-by: Prafulla Mahindrakar <[email protected]>

pmahindrakar-oss approved these changes Jun 18, 2021

View reviewed changes

pmahindrakar-oss merged commit 0ea6da6 into master Jun 18, 2021

pmahindrakar-oss deleted the visualization branch June 18, 2021 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Visualize graphs on command line #75

Visualize graphs on command line #75

kumare3 commented May 28, 2021 •

edited

Loading

jsonporter commented Jun 1, 2021

katrogan Jun 2, 2021

kumare3 Jun 2, 2021

kumare3 Jun 2, 2021

katrogan Jun 2, 2021

yindia commented Jun 3, 2021 •

edited

Loading

kumare3 commented Jun 3, 2021

yindia commented Jun 3, 2021 •

edited

Loading

kumare3 commented Jun 3, 2021

kumare3 commented Jun 3, 2021

yindia commented Jun 3, 2021

yindia commented Jun 5, 2021

kumare3 commented Jun 6, 2021

kumare3 commented Jun 6, 2021

pmahindrakar-oss commented Jun 6, 2021 •

edited

Loading

Visualize graphs on command line #75

Visualize graphs on command line #75

Conversation

kumare3 commented May 28, 2021 • edited Loading

TL;DR

Type

Are all requirements met?

Complete description

jsonporter commented Jun 1, 2021

katrogan Jun 2, 2021

Choose a reason for hiding this comment

kumare3 Jun 2, 2021

Choose a reason for hiding this comment

kumare3 Jun 2, 2021

Choose a reason for hiding this comment

katrogan Jun 2, 2021

Choose a reason for hiding this comment

yindia commented Jun 3, 2021 • edited Loading

kumare3 commented Jun 3, 2021

yindia commented Jun 3, 2021 • edited Loading

kumare3 commented Jun 3, 2021

kumare3 commented Jun 3, 2021

yindia commented Jun 3, 2021

yindia commented Jun 5, 2021

kumare3 commented Jun 6, 2021

kumare3 commented Jun 6, 2021

pmahindrakar-oss commented Jun 6, 2021 • edited Loading

kumare3 commented May 28, 2021 •

edited

Loading

yindia commented Jun 3, 2021 •

edited

Loading

yindia commented Jun 3, 2021 •

edited

Loading

pmahindrakar-oss commented Jun 6, 2021 •

edited

Loading