recommenders-team · miguelgfierro · Dec 20, 2019 · Aug 30, 2019 · Aug 30, 2019 · Aug 30, 2019
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -12,4 +12,5 @@
 <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
 - [ ] I have followed the [contribution guidelines](../CONTRIBUTING.md) and code style for this project.
 - [ ] I have added tests covering my contributions.
-- [ ] I have updated the documentation accordingly.
+- [ ] I have updated the documentation accordingly.
+- [ ] This PR is being made to `staging` and not `master`.
diff --git a/README.md b/README.md
@@ -98,13 +98,14 @@ The following tests run on a Windows and Linux DSVM daily. These machines run 24
 
 | Build Type | Branch | Status |  | Branch | Status |
 | --- | --- | --- | --- | --- | --- |
-| **Linux CPU** | master | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly?branchName=master)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=4792) | | staging | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_staging?branchName=staging)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=4594) |
-| **Linux GPU** | master | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_gpu?branchName=master)](https://msdata.visualstudio.com/DefaultCollection/AlgorithmsAndDataScience/_build/latest?definitionId=4997) | | staging | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_gpu_staging?branchName=staging)](https://msdata.visualstudio.com/DefaultCollection/AlgorithmsAndDataScience/_build/latest?definitionId=4998) |
-| **Linux Spark** | master | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_spark?branchName=master)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=4804) | | staging | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/Recommenders/nightly_spark_staging)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=5186) |
-| **Windows CPU** | master | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_win?branchName=master)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=6743) | | staging | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_staging_win?branchName=staging)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=6752) |
-| **Windows GPU** | master | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_gpu_win?branchName=master)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=6756) | | staging | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_gpu_staging_win?branchName=staging)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=6761) |
-| **Windows Spark** | master | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_spark_win?branchName=master)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=6757) | | staging | [![Status](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_apis/build/status/nightly_spark_staging_win?branchName=staging)](https://msdata.visualstudio.com/AlgorithmsAndDataScience/_build/latest?definitionId=6754) |
-
+| **Linux CPU** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/linux-tests/dsvm_nightly_linux_cpu?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=67&branchName=master) | | staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/linux-tests/dsvm_nightly_linux_cpu?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=67&branchName=staging) |
+| **Linux GPU** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/linux-tests/dsvm_nightly_linux_gpu?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=85&branchName=master) | | staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/linux-tests/dsvm_nightly_linux_gpu?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=85&branchName=staging) |
+| **Linux Spark** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/linux-tests/dsvm_nightly_linux_pyspark?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=86&branchName=master) | | staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/linux-tests/dsvm_nightly_linux_pyspark?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=86&branchName=staging) |
+| **Windows CPU** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/windows-tests/dsvm_nightly_win_cpu?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=101&branchName=master) | | staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/windows-tests/dsvm_nightly_win_cpu?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=101&branchName=staging) |
+| **Windows GPU** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/windows-tests/dsvm_nightly_win_gpu?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=102&branchName=master) | | staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/windows-tests/dsvm_nightly_win_gpu?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=102&branchName=staging) |
+| **Windows Spark** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/windows-tests/dsvm_nightly_win_pyspark?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=103&branchName=master) | | staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/windows-tests/dsvm_nightly_win_pyspark?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=103&branchName=staging) |
+
+<!-- Hiding AzureML build status for the moment
 ### AzureML Build Status
 
 The following tests run on an AzureML [compute target](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-compute-target). AzureML allows to programmatically start a virtual machine, execute the tests, gather the results in [Azure DevOps](https://azure.microsoft.com/en-gb/services/devops/) and shut down the machine.
@@ -113,9 +114,9 @@ The following tests run on an AzureML [compute target](https://docs.microsoft.co
 | --- | --- | --- | --- | --- | --- |  
 | **nightly_cpu_tests** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/nightly_cpu_tests?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=25&branchName=master) |   | Staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/nightly_cpu_tests?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=25&branchName=staging) | 
 | **nightly_gpu_tests** | master | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/bp-nightly_gpu_tests?branchName=master)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=5&branchName=master) |   | Staging | [![Build Status](https://dev.azure.com/best-practices/recommenders/_apis/build/status/bp-nightly_gpu_tests?branchName=staging)](https://dev.azure.com/best-practices/recommenders/_build/latest?definitionId=5&branchName=staging) |
+-->
 
 ### Related projects
 
-
-[Microsoft AI Labs Github](https://aka.ms/ai-labs) Find other Best Practice projects, and Azure AI design patterns in our central repository. 
+[Microsoft AI Github](https://github.com/microsoft/ai): Find other Best Practice projects, and Azure AI design patterns in our central repository. 
 
diff --git a/SETUP.md b/SETUP.md
@@ -81,9 +81,73 @@ To install the PySpark environment:
     python scripts/generate_conda_file.py --pyspark
     conda env create -f reco_pyspark.yaml
 
-Additionally, if you want to test a particular version of spark, you may pass the --pyspark-version argument:
+> Additionally, if you want to test a particular version of spark, you may pass the --pyspark-version argument:
+>
+>     python scripts/generate_conda_file.py --pyspark-version 2.4.0
+
+Then, we need to set the environment variables `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` to point to the conda python executable.
+
+Click on the following menus to see details:
+<details>
+<summary><strong><em>Linux or MacOS</em></strong></summary>
+
+To set these variables every time the environment is activated, we can follow the steps of this [guide](https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux).
+First, get the path of the environment `reco_pyspark` is installed:
+
+    RECO_ENV=$(conda env list | grep reco_pyspark | awk '{print $NF}')
+
+Then, create the file `$RECO_ENV/etc/conda/activate.d/env_vars.sh` and add:
+
+    #!/bin/sh
+    RECO_ENV=$(conda env list | grep reco_pyspark | awk '{print $NF}')
+    export PYSPARK_PYTHON=$RECO_ENV/bin/python
+    export PYSPARK_DRIVER_PYTHON=$RECO_ENV/bin/python
+    export SPARK_HOME_BACKUP=$SPARK_HOME
+    unset SPARK_HOME
+
+This will export the variables every time we do `conda activate reco_pyspark`.
+To unset these variables when we deactivate the environment,
+create the file `$RECO_ENV/etc/conda/deactivate.d/env_vars.sh` and add:
+
+    #!/bin/sh
+    unset PYSPARK_PYTHON
+    unset PYSPARK_DRIVER_PYTHON
+    export SPARK_HOME=$SPARK_HOME_BACKUP
+    unset SPARK_HOME_BACKUP
+
+</details>
 
-    python scripts/generate_conda_file.py --pyspark-version 2.4.0
+<details><summary><strong><em>Windows</em></strong></summary>
+
+To set these variables every time the environment is activated, we can follow the steps of this [guide](https://conda.io/docs/user-guide/tasks/manage-environments.html#windows).
+First, get the path of the environment `reco_pyspark` is installed:
+
+    for /f "delims=" %A in ('conda env list ^| grep reco_pyspark ^| awk "{print $NF}"') do set "RECO_ENV=%A"
+
+Then, create the file `%RECO_ENV%\etc\conda\activate.d\env_vars.bat` and add:
+
+    @echo off
+    for /f "delims=" %%A in ('conda env list ^| grep reco_pyspark ^| awk "{print $NF}"') do set "RECO_ENV=%%A"
+    set PYSPARK_PYTHON=%RECO_ENV%\python.exe
+    set PYSPARK_DRIVER_PYTHON=%RECO_ENV%\python.exe
+    set SPARK_HOME_BACKUP=%SPARK_HOME%
+    set SPARK_HOME=
+    set PYTHONPATH_BACKUP=%PYTHONPATH%
+    set PYTHONPATH=
+
+This will export the variables every time we do `conda activate reco_pyspark`.
+To unset these variables when we deactivate the environment,
+create the file `%RECO_ENV%\etc\conda\deactivate.d\env_vars.bat` and add:
+
+    @echo off
+    set PYSPARK_PYTHON=
+    set PYSPARK_DRIVER_PYTHON=
+    set SPARK_HOME=%SPARK_HOME_BACKUP%
+    set SPARK_HOME_BACKUP=
+    set PYTHONPATH=%PYTHONPATH_BACKUP%
+    set PYTHONPATH_BACKUP=
+
+</details>
 
 </details>
 
@@ -97,68 +161,12 @@ To install the environment:
     python scripts/generate_conda_file.py --gpu --pyspark
     conda env create -f reco_full.yaml
 
+Then, we need to set the environment variables `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` to point to the conda python executable.
+See **PySpark environment** setup section for the details about how to setup those variables.
+where you will need to change `reco_pyspark` string in the commands to `reco_full`.
 </details>
 
 
-> **NOTE** - for PySpark environments (`reco_pyspark` and `reco_full`), we need to set the environment variables
-> `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` to point to the conda python executable.
->
-> Click on the following menus to see details:
->
-> <details>
-> <summary><strong><em>Linux or MacOS</em></strong></summary>
->
-> To set these variables every time the environment is activated, we can follow the steps of this [guide](https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux).
-> Assuming that we have installed the environment in `/anaconda/envs/reco_pyspark`,
-> create the file `/anaconda/envs/reco_pyspark/etc/conda/activate.d/env_vars.sh` and add:
->
->     #!/bin/sh
->     export PYSPARK_PYTHON=/anaconda/envs/reco_pyspark/bin/python
->     export PYSPARK_DRIVER_PYTHON=/anaconda/envs/reco_pyspark/bin/python
->     export SPARK_HOME_BACKUP=$SPARK_HOME
->     unset SPARK_HOME
->
-> This will export the variables every time we do `conda activate reco_pyspark`.
-> To unset these variables when we deactivate the environment,
-> create the file `/anaconda/envs/reco_pyspark/etc/conda/deactivate.d/env_vars.sh` and add:
->
->     #!/bin/sh
->     unset PYSPARK_PYTHON
->     unset PYSPARK_DRIVER_PYTHON
->     export SPARK_HOME=$SPARK_HOME_BACKUP
->     unset SPARK_HOME_BACKUP
-> 
-> </details>
->
-> <details><summary><strong><em>Windows</em></strong></summary>
-> 
-> To set these variables every time the environment is activated, we can follow the steps of this [guide](https://conda.io/docs/user-guide/tasks/manage-environments.html#windows).
-> Assuming that we have installed the environment in `c:\anaconda\envs\reco_pyspark`,
-> create the file `c:\anaconda\envs\reco_pyspark\etc\conda\activate.d\env_vars.bat` and add:
-> 
->     @echo off
->     set PYSPARK_PYTHON=c:\anaconda\envs\reco_pyspark\python.exe
->     set PYSPARK_DRIVER_PYTHON=c:\anaconda\envs\reco_pyspark\python.exe
->     set SPARK_HOME_BACKUP=%SPARK_HOME%
->     set SPARK_HOME=
->     set PYTHONPATH_BACKUP=%PYTHONPATH%
->     set PYTHONPATH=
-> 
-> This will export the variables every time we do `conda activate reco_pyspark`.
-> To unset these variables when we deactivate the environment,
-> create the file `c:\anaconda\envs\reco_pyspark\etc\conda\deactivate.d\env_vars.bat` and add:
-> 
->     @echo off
->     set PYSPARK_PYTHON=
->     set PYSPARK_DRIVER_PYTHON=
->     set SPARK_HOME=%SPARK_HOME_BACKUP%
->     set SPARK_HOME_BACKUP=
->     set PYTHONPATH=%PYTHONPATH_BACKUP%
->     set PYTHONPATH_BACKUP=
-> 
-> </details>
-
-
 ### Register the conda environment as a kernel in Jupyter
 
 We can register our created conda environment to appear as a kernel in the Jupyter notebooks.

diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -14,7 +14,7 @@ WORKDIR ${HOME}
 
 # Install base dependencies
 RUN apt-get update && \
-    apt-get install -y curl git
+    apt-get install -y curl git 
 
 # Install Anaconda
 ARG ANACONDA="https://repo.continuum.io/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh"
@@ -42,7 +42,6 @@ RUN mkdir ${HOME}/.jupyter && \
 # CPU Stage
 FROM base AS cpu
 
-# Setup Conda environment
 RUN python recommenders/scripts/generate_conda_file.py --name base
 
 

diff --git a/notebooks/00_quick_start/README.md b/notebooks/00_quick_start/README.md
@@ -15,7 +15,7 @@ data preparation, model building, and model evaluation by using the utility func
 | [rbm](rbm_movielens.ipynb)| MovieLens | Python CPU, GPU | Utilizing the Restricted Boltzmann Machine (rbm) [4] to predict movie ratings in a Python+GPU (TensorFlow) environment.<br>
 | [rlrmc](rlrmc_movielens.ipynb) | Movielens | Python CPU | Utilizing the Riemannian Low-rank Matrix Completion (RLRMC) [6] to predict movie rating in a Python+CPU environment
 | [sar](sar_movielens.ipynb) | MovieLens | Python CPU | Utilizing Simple Algorithm for Recommendation (SAR) algorithm to predict movie ratings in a Python+CPU environment.
-| [sar_azureml](sar_movielens_with_azureml.ipynb)| MovieLens | Python CPU | An example of how to utilize and evaluate SAR using the [Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/overview-what-is-azure-ml)(AzureML). It takes the content of the [sar quickstart notebook](sar_movielens.ipynb) and demonstrates how to use the power of the cloud to manage data, switch to powerful GPU machines, and monitor runs while training a model.
+| [sar_azureml](sar_movielens_with_azureml.ipynb)| MovieLens | Python CPU | An example of how to utilize and evaluate SAR using the [Azure Machine Learning service](https://docs.microsoft.com/azure/machine-learning/service/overview-what-is-azure-ml) (AzureML). It takes the content of the [sar quickstart notebook](sar_movielens.ipynb) and demonstrates how to use the power of the cloud to manage data, switch to powerful GPU machines, and monitor runs while training a model.
 | [wide-and-deep](wide_deep_movielens.ipynb) | MovieLens | Python CPU, GPU |  Utilizing Wide-and-Deep Model (Wide-and-Deep) [5] to predict movie ratings in a Python+GPU (TensorFlow) environment.
 | [xdeepfm](xdeepfm_criteo.ipynb) | Criteo, Synthetic Data | Python CPU, GPU |  Utilizing the eXtreme Deep Factorization Machine (xDeepFM) [3] to learn both low and high order feature interactions for predicting CTR, in a Python+GPU (TensorFlow) environment.