-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add component-based demand forecasting notebooks (#2470)
* added notebooks * linter * added exceptions, readme and workflow files * changed registries from dev to preview and prod * fixed compute creation step * deleted redundant file. Added try-exccept to avoid the http connection timeout issues * changed gpu compute type due to availability in the test region * added forece rerun setting to pipeline definition * removed forced re-run setting since in the test environment is it triggered by default. * removed repeated experiement name from the HTS nb * added pipeline description to mm and hts nb * removing single model nb and associated files * Removed local data files from the mm nb. Will use data from the public datastore. * modified mm nb to download data from public blob and save as parquet * linter * changed parameter names to be consistent with components' input names in HTS nb * changed parameter names to be consistent with components' input names in MM nb * removed code that enables private features * fixed section reference hyperlinks and removed unused impots from helper scripts * pre-formatted section headers; minor code reformat * added experiment and timout restictions to the MM and HTS nb * added check to make sure all job child runs are posted before downloading forecast results * workround for the PipelinJob bug which is stuck in the preparing state
- Loading branch information
Showing
13 changed files
with
3,136 additions
and
0 deletions.
There are no files selected for viewing
75 changes: 75 additions & 0 deletions
75
...-timeseries-in-pipeline-automl-forecasting-demand-hierarchical-timeseries-in-pipeline.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # This code is autogenerated. | ||
| # Code is generated by running custom script: python3 readme.py | ||
| # Any manual changes to this file may cause incorrect behavior. | ||
| # Any manual changes will be overwritten if the code is regenerated. | ||
|
|
||
| name: sdk-jobs-pipelines-1k_demand_forecasting_with_pipeline_components-automl-forecasting-demand-hierarchical-timeseries-in-pipeline-automl-forecasting-demand-hierarchical-timeseries-in-pipeline | ||
| # This file is created by sdk/python/readme.py. | ||
| # Please do not edit directly. | ||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: "45 5/12 * * *" | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-hierarchical-timeseries-in-pipeline/** | ||
| - .github/workflows/sdk-jobs-pipelines-1k_demand_forecasting_with_pipeline_components-automl-forecasting-demand-hierarchical-timeseries-in-pipeline-automl-forecasting-demand-hierarchical-timeseries-in-pipeline.yml | ||
| - sdk/python/dev-requirements.txt | ||
| - infra/bootstrapping/** | ||
| - sdk/python/setup.sh | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: check out repo | ||
| uses: actions/checkout@v2 | ||
| - name: setup python | ||
| uses: actions/setup-python@v2 | ||
| with: | ||
| python-version: "3.8" | ||
| - name: pip install notebook reqs | ||
| run: pip install -r sdk/python/dev-requirements.txt | ||
| - name: azure login | ||
| uses: azure/login@v1 | ||
| with: | ||
| creds: ${{secrets.AZUREML_CREDENTIALS}} | ||
| - name: bootstrap resources | ||
| run: | | ||
| echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}'; | ||
| bash bootstrap.sh | ||
| working-directory: infra/bootstrapping | ||
| continue-on-error: false | ||
| - name: setup SDK | ||
| run: | | ||
| source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh"; | ||
| source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh"; | ||
| bash setup.sh | ||
| working-directory: sdk/python | ||
| continue-on-error: true | ||
| - name: setup-cli | ||
| run: | | ||
| source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh"; | ||
| source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh"; | ||
| bash setup.sh | ||
| working-directory: cli | ||
| continue-on-error: true | ||
| - name: run jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-hierarchical-timeseries-in-pipeline/automl-forecasting-demand-hierarchical-timeseries-in-pipeline.ipynb | ||
| run: | | ||
| source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh"; | ||
| source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh"; | ||
| bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json"; | ||
| bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" replace_template_values "automl-forecasting-demand-hierarchical-timeseries-in-pipeline.ipynb"; | ||
| [ -f "../../.azureml/config" ] && cat "../../.azureml/config"; | ||
| papermill -k python -p compute_name automl-cpu-cluster automl-forecasting-demand-hierarchical-timeseries-in-pipeline.ipynb automl-forecasting-demand-hierarchical-timeseries-in-pipeline.output.ipynb | ||
| working-directory: sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-hierarchical-timeseries-in-pipeline | ||
| - name: upload notebook's working folder as an artifact | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: automl-forecasting-demand-hierarchical-timeseries-in-pipeline | ||
| path: sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-hierarchical-timeseries-in-pipeline |
75 changes: 75 additions & 0 deletions
75
...ting-demand-many-models-in-pipeline-automl-forecasting-demand-many-models-in-pipeline.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # This code is autogenerated. | ||
| # Code is generated by running custom script: python3 readme.py | ||
| # Any manual changes to this file may cause incorrect behavior. | ||
| # Any manual changes will be overwritten if the code is regenerated. | ||
|
|
||
| name: sdk-jobs-pipelines-1k_demand_forecasting_with_pipeline_components-automl-forecasting-demand-many-models-in-pipeline-automl-forecasting-demand-many-models-in-pipeline | ||
| # This file is created by sdk/python/readme.py. | ||
| # Please do not edit directly. | ||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: "59 7/12 * * *" | ||
| pull_request: | ||
| branches: | ||
| - main | ||
| paths: | ||
| - sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-many-models-in-pipeline/** | ||
| - .github/workflows/sdk-jobs-pipelines-1k_demand_forecasting_with_pipeline_components-automl-forecasting-demand-many-models-in-pipeline-automl-forecasting-demand-many-models-in-pipeline.yml | ||
| - sdk/python/dev-requirements.txt | ||
| - infra/bootstrapping/** | ||
| - sdk/python/setup.sh | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} | ||
| cancel-in-progress: true | ||
| jobs: | ||
| build: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: check out repo | ||
| uses: actions/checkout@v2 | ||
| - name: setup python | ||
| uses: actions/setup-python@v2 | ||
| with: | ||
| python-version: "3.8" | ||
| - name: pip install notebook reqs | ||
| run: pip install -r sdk/python/dev-requirements.txt | ||
| - name: azure login | ||
| uses: azure/login@v1 | ||
| with: | ||
| creds: ${{secrets.AZUREML_CREDENTIALS}} | ||
| - name: bootstrap resources | ||
| run: | | ||
| echo '${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}'; | ||
| bash bootstrap.sh | ||
| working-directory: infra/bootstrapping | ||
| continue-on-error: false | ||
| - name: setup SDK | ||
| run: | | ||
| source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh"; | ||
| source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh"; | ||
| bash setup.sh | ||
| working-directory: sdk/python | ||
| continue-on-error: true | ||
| - name: setup-cli | ||
| run: | | ||
| source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh"; | ||
| source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh"; | ||
| bash setup.sh | ||
| working-directory: cli | ||
| continue-on-error: true | ||
| - name: run jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-many-models-in-pipeline/automl-forecasting-demand-many-models-in-pipeline.ipynb | ||
| run: | | ||
| source "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh"; | ||
| source "${{ github.workspace }}/infra/bootstrapping/init_environment.sh"; | ||
| bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" generate_workspace_config "../../.azureml/config.json"; | ||
| bash "${{ github.workspace }}/infra/bootstrapping/sdk_helpers.sh" replace_template_values "automl-forecasting-demand-many-models-in-pipeline.ipynb"; | ||
| [ -f "../../.azureml/config" ] && cat "../../.azureml/config"; | ||
| papermill -k python -p compute_name automl-cpu-cluster automl-forecasting-demand-many-models-in-pipeline.ipynb automl-forecasting-demand-many-models-in-pipeline.output.ipynb | ||
| working-directory: sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-many-models-in-pipeline | ||
| - name: upload notebook's working folder as an artifact | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: automl-forecasting-demand-many-models-in-pipeline | ||
| path: sdk/python/jobs/pipelines/1k_demand_forecasting_with_pipeline_components/automl-forecasting-demand-many-models-in-pipeline |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.