Skip to content

Commit

Permalink
Documenting Horovod code (flyteorg#496)
Browse files Browse the repository at this point in the history
* Documenting Horovod code

* Added comments

* Updated ml.training.rst

Updated the text

* Introduction, and changes

* Added 'model running locally'

* Update cookbook/case_studies/ml_training/spark_horovod/keras_spark_rossmann_estimator.py

Co-authored-by: Samhita Alla <[email protected]>

* Changes based on review

* Edited 'elapsed time' method

* elapsed time updated

* Update cookbook/case_studies/ml_training/spark_horovod/README.rst

Co-authored-by: Samhita Alla <[email protected]>

* Added more context, explanation

* Added context

* Revert "Merge branch 'SmritiSatyanV' of https://github.com/flyteorg/flytesnacks into SmritiSatyanV"

This reverts commit 9eb2f22a31cee46c1e325b946cac1478289199c3, reversing
changes made to 39ad392f9443f4f9e91a74da2fb3819ccbabb3e1.

* Changes to README.rst

* Update keras_spark_rossmann_estimator.py

* Update keras_spark_rossmann_estimator.py

* Updates based on comments

* Indent fixed

* Added Spark example

* Updates changes based on review

* Linked schedules.rst (from flyte) to this file

* Rolled back house prediction file

The house prediction documentation will be a separate PR, hence the rollback.

* Changes based on review @samhita-alla

* Fixed link @samhita-alla

Note wasn't specified using directives (.. syntax), fixed that

* Fixed links

* Rolled back to changes

* Title changed

* Updated lp_schedules.py

Co-authored-by: Samhita Alla <[email protected]>
  • Loading branch information
SmritiSatyanV and samhita-alla authored Jan 6, 2022
1 parent 4c5d246 commit b4e8b86
Show file tree
Hide file tree
Showing 5 changed files with 144 additions and 60 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -244,4 +244,4 @@ def house_price_predictor_trainer(


# %%
# The output will be a list of house price predictions.
# The output will be a list of house price predictions.
7 changes: 5 additions & 2 deletions cookbook/case_studies/ml_training/spark_horovod/README.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.. _spark_horovod:

Forecast Sales Using Rossmann Store Sales Data Using Horovod and Spark
Forecasting Rossman Store Sales with Horovod and Spark
----------------------------------------------------------------------

The problem statement we will be looking at is forecasting sales using `rossmann store sales <https://www.kaggle.com/c/rossmann-store-sales>`__ data.
Expand Down Expand Up @@ -50,6 +50,9 @@ Since we use the Keras deep learning library, here’s how we install the releva
The installation includes enabling MPI and TensorFlow environments.


.. _flyte-and-spark:

Flyte and Spark
^^^^^^^^^^^^^^^

Expand All @@ -67,7 +70,7 @@ To install the Spark plugin on Flyte, we use the following command:
:alt: Flyte-Spark plugin

Flyte-Spark plugin

The plugin requires configuring the Flyte backend as well. Refer to :ref:`Kubernetes Spark Jobs <plugins-spark-k8s>` for setup instructions.
In a nutshell, here’s how Horovod-Spark-Flyte can be beneficial:
Horovod provides the distributed framework, Spark enables extracting, preprocessing, and partitioning data,
Flyte can stitch the former two pieces together, e.g., by connecting the data output of a Spark transform to a training system using Horovod while ensuring high utilization of GPUs!
Loading

0 comments on commit b4e8b86

Please sign in to comment.