Skip to content

Architecture

Jump to bottom
Oliver Sanders edited this page Nov 8, 2022 · 30 revisions

Diagrams about Cylc UI architecture. Useful for other developers, not intended for wider audience.

Building blocks

The following diagram illustrates the stack used to build Cylc UI. The main library, although not highlighted in the figure, is Vue. Cylc UI is an ES6 Vue application, built with vue-cli.

NOTE: normally you would be working with the webpack, rollerjs, etc, configuration files. But with vue-cli the configuration is stored in vue.config.js and vue-cli will execute webpack/babel/etc for you. The configuration can be kept in vue.config.js (e.g. we don't have webpack elsewhere) or in an external file (e.g. we have babel configuration in babel.config.js). It's up to the developers to choose.

building-blocks

It is easier to read the diagram from the bottom up.

text69374

At the bottom of the stack we have the transpiler Babel, which does things like rewriting const nodeId = node.node?.id (null coalescing operator, not supported in browsers) to something like const nodeId = (node.node) ? node.node.id : undefined. It also transpiles code from libraries like Vuetify (see babel.config.js).

webpack is the bundler used by our code. It coordinates babel, sass, postcss, etc., producing the final files of our application.

It is worth opening the GitHub README.md of each of these building tools, if not already familiar with any of them.

text87363

In the middle of the diagram we have libraries. At the top of the libraries panel we have Vue libraries such as Vue, VueRouter (navigation), Vuex (UI data store), and Vuetify (UI framework). These are closely related to Vue. Next we have utility libraries such as lodash (utilities not found in the JS language), graphql (parsing & merging queries), enumify (because ES6 doesn't have enums), and @mdi/js (material design icons for JS). The penultimate row from the top has just mitt (event bus) and Lumino (tabbed layout). Finally, we have the two libraries used for communication. Axios is used for REST (e.g. JupyterHub REST API) and ApolloClient is used for GraphQL (e.g. GraphiQL, mutations, subscriptions.)

At the top of the stack we have the UI components. We used Vuetify to bootstrap the UI development, slowly adding custom components. On the left hand side you have the Vue components we wrote. Some of these components still use Vuetify components and styling (i.e. css).

rect106110

To the right you have a few Vuetify components that are commonly used.

NOTE: you may not find the imports of Vuetify components, but that's because we have vuetify-loader, which takes care to create the imports on-the-fly as needed. With this, you can simply write <v-list> in your template and the VList will be automagically imported. In theory it should support tree-shaking too, but both Vuetify and Lodash were never very well trimmed.

Layout of Components in the UI

This diagram shows an example of how the UI components are organized when you access the route /#/ (the application dashboard) in your browser. Take a look at src/router/index.js to see what else VueRouter is doing. And for the layout, look at the App.vue computed variables.

cylc-ui-1

The VMain component is not in our source code directory. It was left intentionally to show that Alert.vue and Dashboard.vue both share a parent, and also to point that we have several components in the UI structure that are imported from Vuetify.

The best way to visualize the complete list of components and how they are organized hierarchically is using the Vue Dev Utils browser extension. Note in the Vue Dev Utils screenshot below that it displays the same structure above, but with more components.

image

NOTE: there are two Vue extensions in the screenshot above, one for Vue 2 and one for Vue 3. Pay attention to which version you install. Cylc UI is built with Vue 2.

The UI data store

The Cylc "views" (e.g. Tree, Graph, Table, etc) get their data from the "central data store". This data store is maintained by the "workflow service" and kept up to date with each update as it is received.

The store is implemented with Vuex and stores it's information in a tree:

* user
  * workflow-part
    * workflow
      * cycle
        * task
          * job

Additionally there are three indexes prefixed with a $ dollar sign:

  • $index Stored at the top level of the tree, this contains a mapping of every ID stored to its node in the tree.
  • $namespaces Stored on workflow nodes, this is populated with Task nodes (aka TaskDef objects in cylc-flow).
  • $edges Stored on workflow nodes, this is populated with Edge nodes (aka graph edges).

Finally there is a secondary tree present in cycle nodes which contains the family inheritance structure called familyTree.

To filter nodes from the store use the getNodes getter. If your view requires different sorting or data structure, use computed properties to reactively reshape the data to fit. If computed methods are too difficult to manage, e.g. for efficiency reasons, the view can register a local callback enabling it to intercept incoming data in order to maintain its own local store. Note this should be a last resort.

To understand how Vuex works, what are states, actions, and mutations, it is recommended the reading of the following Vuex docs sections:

  1. Introduction - Getting Started
  2. Core Concepts - State
  3. Core Concepts - Getters
  4. Core Concepts - Mutations
  5. Core Concepts - Actions
  6. Core Concepts - Modules

UI Views

This section revolves around the Workflow.vue view and the WorkflowService. That view utilizes the Lumino.vue component to show other views in tabs.

NOTE: Cylc 7 had the concept of Views, like the Tree View, the Graph View, the Dot View, and so it goes. In Cylc 8 UI, the Vue framework has the concept of VueRouter's views. To add to the confusion, the Cylc 7 Tree View is implemented as a Vue component, that is used in a VueRouter view. So note that these terms may be used interchangeably.

The WorkflowService drives the subscription for a Cylc UI View.

In Cylc 8 UI, the Views are VueRouter views (although Vue components are also supported) and use Vue mixins to define the expected behavior of Views. The mixins will trigger functions in the WorkflowService to start and stop subscriptions.

Not all Cylc UI views need to have a GraphQL subscription. The Mutation view, for instance, does not have one. If a view does not need GraphQL, it will require fewer mixins.

A view like the Tree View, that contains one GraphQL subscription to fetch data to build the tree, will use the following mixins:

  • mixins/index for the page title (this is for when the view is accessed directly)
  • mixins/graphql to define the GraphQL subscription variables (e.g. workflowId: some/workflow/run1)
  • mixins/subscriptionComponent hooks up the Vue component lifecycle hooks with WorkflowService functions
  • mixins/subscriptionView hooks up the VueRouter navigation guards with WorkflowService functions
  • mixins/subscription this mixins is used by the two subscription* mixins above, and adds the ViewState state and setAlert function to the view

bitmap

Each Cylc UI View has a ViewState that may hold the state of NO_STATE, LOADING, ERROR, or COMPLETE. The mixins/subscription has a computed isLoading property that uses the ViewState to return a boolean. The WorkflowService updates the ViewState.

Finally, views (or components) also need to define the query property, in either data or computed sections. This property must have the type model/SubscriptionQuery. A SubscriptionQuery contains:

  • A GraphQL Query (e.g. subscription OnWorkflowTreeDeltasData ($workflowId: ID) { ... }, see graphql/queries.js)
  • An object with the GraphQL variables (probably computed via the mixins/graphql, may be empty e.g. GScan query)
  • A name (GScan uses a query named root, while the TreeView uses workflow)
  • A list of action names or callbacks (Vuex action/mutation list executed for each GraphQL response message)
  • A list of tear down action names (for when the View or component is removed/destroyed, to clear the UI store, etc)

NOTE: There is a pull request that removes the action and tear down actions

When you navigate to a URL like /#/tree/:workflowId the Cylc Tree View will be a VueRouter view. It will use its navigation guards beforeRouteEnter to start a GraphQL subscription using the Tree view query property. When you type a new URL in your browser like /#/tree/:anotherWorkflowId Vue will re-use the component, updating it, but VueRouter view will update the subscription too via the beforeRouteUpdate navigation guard. Finally, when you leave the VueRouter view, then it will stop the GraphQL subscription (via the Vue component lifecycle function beforeDestroy.)

When you navigate to a URL like /#/workflows/:workflowId the Cylc Workflow View will be a VueRouter View. It will also use the navigation guards to tell the WorkflowService to start the subscriptions. However, the Workflow View does not have a query property. Instead, it uses the JupyterLab Lumino component functions to capture when a tab with a View is added or removed, calling the functions to update the subscriptions. So the Lumino actions mimic what the navigation guards and Vue component lifecycle functions do.

bitmap2

Using Lumino tabs allows you to have multiple Cylc Views in the same VueRouter view. These views may be using different GraphQL queries, but for performance we try to merge the queries when possible. The details on how these queries' subscriptions are handled are detailed in the next section.

NOTE: See this issue for pending improvements for the Cylc Views

GraphQL subscriptions

Cylc UI uses ApolloClient to handle GraphQL queries. In GraphQL, you can pass a query and “watch” for changes to that query. That is known as a Subscription. The only difference being whether you execute a GraphQL query that starts with query workflows { ... } or one that starts with subscription workflows { ... }.

Not all servers support GraphQL subscription. In fact, Cylc UIS did not support subscriptions initially. That was added in the backend later. Similarly, not all clients support GraphQL subscriptions. We added it later too, by modifying the transport link, using subscriptions-transport-ws.

The piece of code responsible to bridge the UI and the UIS is the WorkflowService. It is a singleton in the UI, exported to the Vue prototype as $workflowService. I.e. in any instance of Vue, like a component or a VueRouter view, you can access the service as .$workflowService.

WorkflowService is responsible for:

  • maintaining a list of subscriptions and subscribers (a new subscription will be pending to be started)
  • starting pending subscriptions
  • stop subscriptions without subscribers
  • merging queries of subscriptions with multiple subscribers (i.e. many subscribers with a query that shares a common root)

bitmap

Each subscriber of the WorkflowService will provide a query that contains a name. The name is the key for letting the WorkflowService whether a query must be merged or not.

GScan, for instance, uses the query named root. It is a GraphQL subscription that queries all existing workflows to be displayed in the UI sidebar. Tree, on the other hand, uses a query called workflow and that is limited to querying the UIS about a single workflow data.

Queries use GraphQL fragments for simplicity and re-usability. When merging queries, we do not use fragments directly. We will parse the final query, and combine both parsed queries creating a new one. Using fragments in similar queries, however, makes it easier to maintain the GraphQL queries and reduces the risk of bugs due to wrong data requested.

The merging process will raise issues if it cannot merge the queries for many reasons, like:

  • incompatible types (you have an argument in one query that is an Object, but in the other query it is an Array)
  • duplicate fragments (that is not valid for GraphQL)
  • multiple documents (i.e. you have an invalid GraphQL query, with two subscriptions for instance)
  • you used an InlineFragment (not supported, but can be added later)
  • you tried to combine a query and a subscription
  • you tried to combine a query and a fragment
  • you used different enums (we are not able to merge them at the moment)
  • you have different variables (to avoid issues where you have 2 queries for 1 workflow, but you specified two different workflow IDs in the variables, for instance)
  • you have a GraphQL value that is not supported (we support native types like ints/floats/text, Objects, nulls, List values, etc)
  • ...

NOTE: you can debug the WorkflowService to see how it merges queries. If you need to debug the deltas too, take a look at this Development note

When data is received from a GraphQL subscription, the WorkflowService will then call the callbacks of each subscribers passing the received data to the callback. From that point forward, it will probably process the data and update the UI store. Components will react to the changes to the store data, and the UI will appear to be magically updated :)

Clone this wiki locally