Google Summer of Code 2021 Project

_posts/2021-08-23-gsoc-2021-project.md

@@ -0,0 +1,208 @@
+---
+layout: post
+title: Google Summer of Code 2021 - Dask Project
+author: Freyam Mehta and Genevieve Buckley
+theme: twitter
+---
+
+{% include JB/setup %}
+
+## Executive Summary
+
+Here's an update on new features related to visualizing Dask graphs and HTML representations. You can try these new features today with version `2021.08.1` or above. This work was done by Freyam Mehta during the Google Summer of Code 2021. Dask took part in the program under the NumFOCUS umbrella organization.
+
+## Contents
+
+- [Visualizing Dask graphs](#visualizing-dask-graphs)
+    - [Graphviz node size scaling](#graphviz-node-size-scaling)
+    - [New tooltips](#new-tooltips)
+    - [Color by layer type](#color-by-layer-type)
+    - [Bugfix in visualize method](#bugfix-in-visualize-method)
+- [HTML Representations](#html-representations)
+    - [Array images in HTML repr for high level graphs](#array-images-in-html-repr-for-high-level-graphs)
+    - [New HTML repr for ProcessInterface class](#new-html-repr-for-processinterface-class)
+    - [New HTML repr for Security class](#new-html-repr-for-security-class)
+
+## Visualizing Dask graphs
+
+There are several new features involving Dask graph visualization. Freyam worked on making them more illustrative, engaging, and informative. The Graphviz library boasts a great set of attributes which can be modifified to create a more visually appealing output. 
+
+These features primarily improve the Dask high level graph visualizations. Both low level and high level Dask graphs can be accessed with very similar methods:
+
+- Dask low level graph: `result.visualize()`
+- Dask high level graph: `result.dask.visualize()`
+
+### Graphviz node size scaling
+
+The first change you may notice to the Dask high level graphs, is that the node sizes have been adjusted to scale with the number of tasks in each layer. Layers with more tasks would appear larger than the rest.
+
+This is a helpful feature to have, because now users can get a much more intuitive sense of where the bulk of their computation takes place.
+
+Example:
+
+```python
+import dask.array as da
+
+array = da.random.random((10000, 10000), chunks=(200, 200))
+result = array + array.T - array.mean(axis=0)
+
+result.dask.visualize()  # Dask high level graph
+```
+
+<img src="/images/gsoc21/7869.png" alt="Example: graphviz node size scaling, pull request #7869" height=414 width=736>
+
+Note: this change only affects the graphviz output for Dask high level graphs. Low level graphs are left unchanged.
+
+Reference: [Pull request #7869 by Freyam Mehta *"Add node size scaling to the Graphviz output for the high level graphs"*](https://github.com/dask/dask/pull/7869)
+
+### New tooltips
+
+Dask high level graphs now include hover tooltips to provide a brief summary of more detailed information. To use the tooltips, generate a dask high level graph (eg: `result.dask.visualize()`) then hover your mouse above the layer you are interested in.
+
+<img src="/images/gsoc21/7973.png" alt="Example: tooltips provide extra information, pull request #7973" height=414 width=736>
+
+Tooltips provide information such as the layer type and number of tasks associated with it. There is additional information provided for specific dask collections, like dask arrays and dataframes.
+
+Dask array tooltip information additionally includes:
+* Array shape
+* Chunk size
+* Chunk type (eg: are the array chunks numpy, cupy, sparse, etc.)
+* Data type (eg: are the array values float, integer, boolean, etc.)
+
+Dask dataframe tooltip information additionally includes:
+* Number of partitions
+* Dataframe type
+* Dataframe columns
+
+Users have asked for a less overwhelming view into the dask task graph. We hope the high level graph view coupled with more detailed tooltip information can provide this middle ground, with enough information to be useful, but not so much as to become overwhelming (like the low level task graphs for large computations).
+
+Note: This feature is available for SVG output. Other image formats, like `.png`, etc. do not support tooltips.
+
+Reference: [Pull request #7973 by Freyam Mehta *"Add tooltips to graphviz"*](https://github.com/dask/dask/pull/7973)
+
+### Color by layer type
+
+There is also a new feature enabling users to color code a high level graph according to layer type. This option can be enabled by passing the `color="layer_type"` keyword argument, eg: `result.dask.visualize(color="layer_type")`. This change is intended to make it easier for users to see which layer types predominate.
+
+While there are no hard and fast rules about what makes a Dask computation efficient, there are some general guidelines:
+
+* Dataframe shuffles are particularly expensive operations. You can [read more about this here](https://docs.dask.org/en/latest/dataframe-best-practices.html#avoid-full-data-shuffling).
+* Reading and writing data to/from disk can be a limiting factor. (Similarly, you want to avoid Dask spilling data to disk wherever possible - monitor the worker memory plot on the dask dashboard if you suspect this may be happening).
+* Blockwise layers are generally efficient for computation.
+
+See the [Dask best pracices](https://docs.dask.org/en/latest/best-practices.html) pages for more information on creating more efficient Dask computations.
+
+Example:
+
+```python
+# code
+```
+
+<img src="/images/gsoc21/7974.png" alt="Example: Dask graph colored by layer type, pull request #7974" height=414 width=736>
+
+Reference: [Pull request #7974 by Freyam Mehta *"Add colors to represent high level layer types"*](https://github.com/dask/dask/pull/7974)
+
+### Bugfix in visualize method
+
+Freyam also fixed a bug which caused an error when users tried to call `dask.visualize()` with `filename=None` (issue [#7685](https://github.com/dask/dask/issues/7685), fixed by pull request [#7740](https://github.com/dask/dask/pull/7740)).
+
+The bug was fixed by adding an extra condition before it reaches the error. If the format is `None`, Dask now uses use a default `png` format.
+
+```python
+import dask
+import dask.array as da
+
+array = da.arange(10)
+dask.visualize(array, filename=None)  # success
+```
+
+Reference: [Pull request #7740 by Freyam Mehta *"Fixing calling .visualize() with filename=None"*](https://github.com/dask/dask/pull/7740)
+
+## HTML representations
+
+Dask makes use of HTML representations in several places, for example in Dask collections like the Array and Dataframe classes (for background reading, see [this blogpost](https://matthewrocklin.com/blog/2019/07/04/html-repr)).
+
+More recently, we've introduced HTML representations for high level graphs into Dask, and Jacob Tomlinson has implemented HTML representations in several places in the dask distributed library (for further reading, see [this other blogpost](https://blog.dask.org/2021/07/07/high-level-graphs#visualization)).
+
+During Freyam's Google Summer of Code project, he extended the HTML representations for Dask high level graphs to include images, and introduced two entirely new HTML representations to the dask distributed library.
+
+### Array images in HTML repr for high level graphs
+
+The HTML representation for dask high level graphs has been extended, and now includes SVG images of dask arrays at intermediate stages of computation.
+
+The motivation for this feature is similar to the motivation behind adding tooltips, discussed above. Users want easier ways to access information about the way a Dask computation changes as it moves through each stage of computation. We hope this improvement to the HTML representation for Dask high level graphs will provide an at a glance summary of array shape and chunk size at each stage.
+
+Example:
+
+```python
+import dask.array as da
+
+array = da.ones((10, 20), chunks=(5, 10))
+array = array.T
+
+array.dask  # shows the HTML representation in Jupyter
+```
+
+<img src="/images/gsoc21/7886.png" alt="Example: Array images now included in HTML representation of Dask high level graphs, pull request #7886" height=414 width=736>
+
+Reference: [Pull request #7886 by Freyam Mehta *"Add dask.array SVG to the HTML Repr"*](https://github.com/dask/dask/pull/7886)
+
+### New HTML repr for ProcessInterface class
+
+A new HTML representation has been created for the `ProcessInterface` class in [dask distributed](https://github.com/dask/distributed/). The `ProcessInterface` class is not intended to be used directly. Instead, this information should be accessed via SSH scheduler or workers.
+
+The HTML representation displays the status, address, and external address of the process.
+
+There are three possible status options:
+* Process created, not yet running (blue icon)
+* Process is running (green icon)
+* Process closed (orange icon)
+
+Example:
+
+```python
+from dask.distributed import LocalCluster, Client, SSHCluster
+
+cluster = SSHCluster(["127.0.0.1", "127.0.0.1", "127.0.0.1"])
+cluster.scheduler  # HTML representation for the SSH scheduler, shown in Jupyter
+# or
+cluster.workers[0]  # HTML representation for the first SSH worker in the cluster
+```
+
+<img src="/images/gsoc21/5181.png" alt="Example: New HTML representation for distributed ProcessInterface class, pull request #5181" height=414 width=736>
+
+Reference: [Pull request #5181 by Freyam Mehta *"Add HTML Repr for ProcessInterface Class and all its subclasses"*](https://github.com/dask/distributed/pull/5181)
+
+### New HTML repr for Security class
+
+Pull request [#5178](https://github.com/dask/distributed/pull/5178) added a new HTML representation for the `Security` class in the [dask distributed library](https://github.com/dask/distributed/). In addition, the text representation has also been updated to reflect the same information shown in the HTML representation.
+
+
+The `Security` HTML representation shows:
+* Whether encryption is required
+* Whether the object instance was created using `Security.temporary()` or `Security(**paths_to_keys)`.
+    * For temporary security objects, keys are generated dynamically and the only copy is kept in memory.
+    * For security objects created using keys stored on disk, the HTML representation will show the full filepath to the relevant security certificates on disk.
+
+
+Example: temporary security object
+
+```python
+from dsak.distributed import Security
+
+s = Security.temporary()
+s  # shows the HTML representation in Jupyter
+```
+
+Example: security object using certificates saved to disk
+
+```python
+from dsak.distributed import Security
+
+s = Security(require_encryption=True, tls_ca_file="ca.pem", tls_scheduler_cert="scert.pem")
+s  # shows the HTML representation in Jupyter
+```
+
+<img src="/images/gsoc21/5178.png" alt="Example: New HTML representation for distributed Security class, pull request #5178" height=414 width=736>
+
+Reference: [Pull request #5178 by Freyam Mehta *"Add HTML Repr for Security Class"*](https://github.com/dask/distributed/pull/5178/)