From e10b10c75b44539a51830690d2eb4252d01f4b2e Mon Sep 17 00:00:00 2001 From: Christian Lechner Date: Mon, 20 Sep 2021 20:48:41 +0200 Subject: [PATCH] Extended hello-sample (#73) * extended hello-sample with additional samples * corrections+remarks from review --- docs/hello-sample.md | 337 +++++++++++++++++++++++++++++++------------ 1 file changed, 243 insertions(+), 94 deletions(-) diff --git a/docs/hello-sample.md b/docs/hello-sample.md index 0f6a6eed..bafd7248 100644 --- a/docs/hello-sample.md +++ b/docs/hello-sample.md @@ -1,150 +1,302 @@ -# Running the 'Hello' Sample +# Running the 'Hello' Samples -Here are simple instructions for running the 'Hello' sample project. We included scripts that make it easy to build, run, and deploy this application. Also, this sample is a great resource for understanding how Durable Functions and Netherite work, and an excellent starting point for creating your own projects. +In order to speed up your first experience with Netherite we provide some samples that serve this goal. They are also a great resource for understanding how Durable Functions and Netherite work. In addition you can use them as a starting point for creating your own projects. + +All samples are in the folder [/samples/](https://github.com/microsoft/durabletask-netherite/tree/main/samples/). We provide samples for the following languages: + +- C#/.NET Core 3.1 in the folder [/samples/Hello_Netherite_with_DotNetCore](https://github.com/microsoft/durabletask-netherite/tree/main/samples/Hello_Netherite_with_DotNetCore) +- TypeScript/Node.js in the folder [/samples/Hello_Netherite_with_TypeScript](https://github.com/microsoft/durabletask-netherite/tree/main/samples/Hello_Netherite_with_TypeScript) +- Python in the folder [/samples/Hello_Netherite_with_Python](https://github.com/microsoft/durabletask-netherite/tree/main/samples/Hello_Netherite_with_Python) + +In addition we provide several PowerShell scripts that enable you to setup the necessary resources on Azure as well as to build, run, and deploy the sample applications. You find them in the folder [/samples/scripts](https://github.com/microsoft/durabletask-netherite/tree/main/samples/scripts) ## Prerequisites +### General Prerequisites + First, make sure you have the following installed on your machine: -1. [.NET Core SDK 3.1](https://dotnet.microsoft.com/download/dotnet-core/3.1) or later. -You can run `dotnet --list-runtimes` to check what is installed. -2. [Azure Functions Core Tools 3.x](https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=windows%2Ccsharp%2Cbash) or later. -You can run `func --version` to check what is installed. -3. [The Azure CLI 2.18.0](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) or later. -You can run `az --version` to check what is installed. -4. [PowerShell 7.1](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell?view=powershell-7.1) or later. -You can check what is installed with `pwsh --version`. PowerShell is easy to install or update via `dotnet tool install --global PowerShell` or `dotnet tool install --global PowerShell`, respectively. -Also, you need an Azure subscription, to allocate and deploy the required resources. +1. [.NET Core SDK 3.1](https://dotnet.microsoft.com/download/dotnet-core/3.1) or later. You can run `dotnet --list-runtimes` to check what is installed. + +2. [Azure Functions Core Tools 3.x](https://docs.microsoft.com/azure/azure-functions/functions-run-local?tabs=windows%2Ccsharp%2Cbash) or later. You can run `func --version` to check what is installed. + +3. [PowerShell 7.1](https://docs.microsoft.com/powershell/scripting/install/installing-powershell?view=powershell-7.1) or later. You can check what is installed with `pwsh --version`. PowerShell is easy to install or update via `dotnet tool install --global PowerShell` or `dotnet tool install --global PowerShell`, respectively. + +4. [Azurite 3.x](https://github.com/Azure/Azurite) as storage emulator for the local execution. You can run `azurite --version` to check what is installed. + +5. [The Azure CLI 2.18.0](https://docs.microsoft.com/cli/azure/install-azure-cli) or later. You can run `az --version` to check what is installed. + +6. [An Azure subscription](https://azure.microsoft.com/free/search/), to allocate and deploy the required resources. + +To execute the REST calls we recommend to use the [RESTClient for VSCode](https://marketplace.visualstudio.com/items?itemName=humao.rest-client). + +📝 **Remark:** You can execute Durable Functions with Netherite local-only without an Azure subscription. In that case you are fine with installing the prerequisites 1. - 4. + +### Language-Specific Prerequisites + +Depending on the language you want to use, you have some more things to install. You find them in the following sections. + +#### C# + +You are already good to go, nothing more to install. + +#### TypeScript + +1. [Node.js](https://nodejs.org/en/download/). Install an LTS version. You can run `node --version` to check what is installed. +2. [TypeScript](https://www.typescriptlang.org/download). You can run `tsc --version` to check what is installed. + +#### Python + +1. [Python](https://www.python.org/downloads/). Check the [documentation](https://docs.microsoft.com/de-de/azure/azure-functions/supported-languages#languages-by-runtime-version) for the currently supported versions. You can run `python --version` to check what is installed. +2. [Python extension for Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=ms-python.python) + +If you are working wth Windows, you can install Python directly form the Microsoft Store e. g. [version 3.8](https://www.microsoft.com/de-de/p/python-38/9mssztt1n39l?rtc=1#activetab=pivot:overviewtab). ## Get it and build it -The sample is in the directory [/samples/hello](https://github.com/microsoft/durabletask-netherite/tree/main/samples/Hello). You can download just this folder if you wish, or clone the entire repository. Note that PowerShell sometimes refuses to execute downloaded files; so using git clone may work better. +The samples are in the directory [/samples/](https://github.com/microsoft/durabletask-netherite/tree/main/samples). You can download just this folder if you wish, or clone the entire repository. Note that PowerShell sometimes refuses to execute downloaded files so using git clone may work better. + +The language-specific folder in the `samples` folder contain "minimal" Azure Durable Functions projects (much like the [DF Quick Start](https://docs.microsoft.com/azure/azure-functions/durable/durable-functions-create-first-csharp?pivots=code-editor-visualstudio)). + +⚠ **CAUTION:** Be aware that all projects have no authentication in place. This makes the calling of the Functions easy, but this is not feasible setup for a production environment. + +The build process depends on the language you want to use. The following sections guide you through the specific steps. + +### C# + +The folder [/samples/Hello_Netherite_with_DotNetCore](https://github.com/microsoft/durabletask-netherite/tree/main/samples/Hello_Netherite_with_DotNetCore) contains the files for the Durable Function App in C#. -The project contains a "minimal" .NET Azure Durable Functions project (much like the [DF Quick Start](https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-create-first-csharp?pivots=code-editor-visualstudio)): -- a single file of code, `HelloCities.cs`, which defines three functions: a trigger, an orchestration, and an activity. -- a configuration file `host.json` which contains settings. -- a configuration file `local.settings.json` which contains settings for local testing. +The main files and folders are: -Additionally, the folder contains six scripts (files ending in *.ps1) that make it easy to perform typical operations (init, run, deploy, clear, delete) using automated CLI commands. These scripts are purely optional; one can perform all the steps in these scripts manually, using the Azure CLI or the Azure Portal. +- the single file of code, `HelloCities.cs`, which defines three functions: a trigger, an orchestration, and an activity. +- the configuration file `host.json` which contains settings. +- the configuration file `local.settings.json` which contains settings for local execution. + +Additionally, the folder contains PowerShell scripts (files ending in `*.ps1`) that make it easy to run and deploy the Function using automated CLI commands. These scripts are optional. You can perform all the steps in these scripts manually, using the Azure CLI or the Azure Portal. To build, open a command shell in that directory and type -```shell + +```powershell dotnet build ``` -You can do this also from within VS code, or Visual Studio, of course. +### TypeScript + +The folder [/samples/Hello_Netherite_with_TypeScript](https://github.com/microsoft/durabletask-netherite/tree/main/samples/Hello_Netherite_with_TypeScript) contains the files for the Durable Function App in TypeScript. + +The main files and folders are: + +- the folders containing the code (`*.ts`) and configuration (`function.json`) of the three functions: the trigger (`DurableFunctionsNetheriteStarter`), the orchestrator (`DurableFunctionsNetheriteOrchestrator`), and the activity (`HelloCityNetherite`). +- the configuration file `host.json` which contains settings. +- the configuration file `local.settings.json` which contains settings for local execution. +- the `extensions.csproj` file that defines the needed extension for Durable Functions and Netherite as we cannot rely on the Extension Bundle mechanism. +- the `package.json` file containing the metadata of the project and the dependencies +- the `tsconfig.json` file containing the compiler options for TypeScript + +Additionally, the folder contains PowerShell scripts (files ending in `*.ps1`) that make it easy to run and deploy the Function using automated CLI commands. These scripts are optional. You can perform all the steps in these scripts manually, using the Azure CLI or the Azure Portal. + +To build, open a command shell in that directory and type + +```powershell +npm install + +func extensions install + +npm run build +``` + +### Python + +The folder [/samples/Hello_Netherite_with_TypeScript](https://github.com/microsoft/durabletask-netherite/tree/main/samples/Hello_Netherite_with_TypeScript) contains the files for the Durable Function App in Python. + +The main files and folders are: + +- the folders containing the code (`*.py`) and configuration (`function.json`) of the three functions: the trigger (`DurableFunctionsNetheriteStarter`), the orchestrator (`DurableFunctionsNetheriteOrchestrator`), and the activity (`HelloCityNetherite`). +- the configuration file `host.json` which contains settings. +- the configuration file `local.settings.json` which contains settings for local execution. +- the `extensions.csproj` file that defines the needed extension for Durable Functions and Netherite as we cannot rely on the Extension Bundle mechanism. +- the `requirements.txt` file containing the dependencies + +Additionally, the folder contains PowerShell scripts (files ending in `*.ps1`) that make it easy to run and deploy the Function using automated CLI commands. These scripts are optional. You can perform all the steps in these scripts manually, using the Azure CLI or the Azure Portal. + +To build, open a command shell in that directory and type + +```powershell +py -m venv .venv + +pip install -r requirements.txt + +func extensions install +``` + +## Run it locally + +We can now give it a first try and run the Functions locally without using resources on Azure. To do so start the Azurite storage emulator (e. g. via the [VS Code plugin](https://marketplace.visualstudio.com/items?itemName=Azurite.azurite)). + +Each language-specific folder contains a pre-configured `local.settings.json` file that contains the fitting parameters for a local execution namely: + +```json +"AzureWebJobsStorage": "UseDevelopmentStorage=true", +"EventHubsConnection": "MemoryF", +``` + +When the storage emulator is up and running you can start the Functions depending on your language of choice. + +### C# + +- go to the directory `bin\Debug\netcoreapp3.1` within the folder `Hello_Netherite_with_DotNetCore`. +- start the Functions runtime with `func start`. + +### TypeScript + +- go to the directory `Hello_Netherite_with_TypeScript`. +- start the functions runtime via `npm run start`. + +### Python + +- go to the directory `Hello_Netherite_with_Python`. +- start the Functions runtime with `func start`. + +### Debug Mode + +You can also start the Functions in debugging mode by opening the language-specific directory in VS Code/Visual Studio and pressing `F5`. You can then also set breakpoints. + +### Call the Function + +As soon as you have started the Function you will see an endpoint in the logs to trigger the Function execution. We have prepared the script `request.http` in [/samples/scripts](https://github.com/microsoft/durabletask-netherite/tree/main/samples/scripts) based on the VS Code REST extension to call the endpoints. + +You can also use Postman or CURL to call the invocation endpoints via a *POST* request: + +- C#: `http://localhost:7071/api/hellocities` +- TypeScript and Python: `http://localhost:7071/api/orchestrators/DurableFunctionsNetheriteOrchestrator` + +This should trigger any breakpoints you have set, and basically produce the following output: + +```json +["Tokyo","Seattle","London"] +``` + +Be aware that you must query the `statusUri` for non.NET languages to get the result including additional status information. + +### Cleanup + +To shut down the functions app, hit `Control^C` in the terminal which runs it. You can also stop Azurite. + +🚀 **Great** - you have successfully executed Durable Functions with Netherite locally. Let us move on and combine this with the necessary resources on Azure. ## Create Azure Resources -First, edit the following line of `settings.ps1` -```PowerShell +First, edit the following line of `settings.ps1` in the folder [/samples/scripts](https://github.com/microsoft/durabletask-netherite/tree/main/samples/scripts) + +```powershell $name="globally-unique-lowercase-alphanumeric-name-with-no-dashes" ``` -and replace the string with a globally unique string (e.g. `sbnethtest391`) for naming the Azure resources. -**CAUTION:** As this string is used for naming the storage account, it must be between 3 and 24 characters in length and may contain numbers and lowercase letters only. +and replace the string with a globally unique string (e. g. `sbnethtest391`) for naming the Azure resources. + +⚠ **CAUTION:** As this string is used for naming the storage account, it must be between 3 and 24 characters in length and may contain numbers and lowercase letters only. You may inspect and change any other settings in `settings.ps1` also, if desired. Next, connect to your Azure subscription: -```shell + +```powershell az login ``` Then, run the `init.ps1` script. This can take some time: -|Windows|Other| -|-------|-----| +|Windows |Other | +|------- |----- | |`pwsh ./init.ps1`|`./init.ps1`| - As it runs, this script creates the following resources: -1. A **storage account** for storing the function application, and the state of orchestrations and entities. -2. An **eventhubs namespace** for providing the persistent queues used by Netherite. -3. A **resource group** that contains the other two resources. + +1. A **resource group** for the resources. +2. A **storage account** for storing the Function application, and the state of orchestrations and entities. +3. An **EventHubs namespace** for providing the persistent queues used by Netherite. You can inspect these resources in the [Azure Portal](https://portal.azure.com), and change their configuration parameters. -**CAUTION:** The EventHubs namespace incurs continuous charges even if not used. Delete them by running `delete.ps1` once you are done with this sample. +⚠ **CAUTION:** The EventHubs namespace incurs continuous charges even if not used. Make sure to clean up resources by running `delete.ps1` once you are done with this sample. -Both the storage account and the EventHubs namespace have a *connection string* that is needed for the application to use them. -Our PowerShell scripts automatically look them up using the CLI. Alternatively, you can manually set the environment variables `AzureWebJobsStorage` and `EventHubsConnection` to contain these connection strings. +Both the storage account and the EventHubs namespace have a *connection string* that is needed for the application to use them. Our PowerShell scripts automatically look them up using the CLI. Alternatively, you can manually set the environment variables `AzureWebJobsStorage` and `EventHubsConnection` to contain these connection strings or put them into the `local.settings.json` file. -## Run or Debug locally +## Run or Debug hybrid -To start the application on your local machine, do *one* of A, B, or C, whichever you prefer: +As we have the needed resources in place we can now execute our Functions using the storage account and the EventHub on Azure. -A. Follow these steps: +You have two options now: - 1. set the environment variables `AzureWebJobsStorage` and `EventHubsConnection` to the respective connection strings. You can look these up in the [Azure portal](portal.azure.com) or with the CLI. - 2. go to the directory `bin\Debug\netcoreapp3.1`. - 2. start the functions runtime with `func start`. +1. Set the environment variables `AzureWebJobsStorage` and `EventHubsConnection` to contain these connection strings or put them into the `local.settings.json` file and start the Function as for the local execution before. +2. Execute the pre-configured scripts in the language-specific folders: + - C#: run-dotnet-function-hybrid.ps1 + - TypeScript: run-typescript-function-hybrid.ps1 + - Python: run-python-function-hybrid.ps1 -B. Run the `run.ps1` script. This is the same as A, but automatic. +You may experience some delay initially, as the taskhub needs to be created the first time it is used. -C. Follow these steps: +Trigger the Function via the HTTP request as in the local execution scenario. - 1. set the environment variables `AzureWebJobsStorage` and `EventHubsConnection` to the respective connection strings. - 2. Open the project in Visual Studio or VS Code and start debugging. You can also set a breakpoint. +### Cleanup -After executing this script successfully, near the bottom of the output, you should see a list of all three published functions: +To shut down the functions app down, you can hit `Control^C` in the terminal which runs it. -```text -Http Functions: +⚠ **CAUTION:** If you leave the app running on your local machine during the next step, it will execute alongside with the cluster of machines you run in the cloud! That is, the Netherite load balancer will distribute the partitions over all machines in the cloud, and your local machine. While this "super-cluster" would function correctly, it is probably not what you want. - HelloCities: [GET,POST] http://localhost:7071/api/hellocities -``` +## Deploy it to Azure -You can now *test the running app* by issuing an HTTP request in a separate terminal: +Now it is time to bring the Function App to Azure and run it in the cloud. The deployment to Azure comprises two steps: -```shell -curl http://localhost:7071/api/hellocities -``` +- Creation of the Function App on Azure +- Deployment of Azure Function to the Function App on Azure. -Which should trigger any breakpoints you have set, and eventually produces the following output, as expected: -``` -["Tokyo","Seattle","London"] -``` +We will walk through these steps in the next two sections. -You may experience some delay initially, as the taskhub needs to be created the first time it is used. +📝 **Remark:** We use a *premium functions plan* for now, until we implement support for the consumption plan. -To shut down the functions app down, you can hit Control^C in the terminal which runs it. +### Setup Azure Functions App on Azure -Note that if you leave the app running on your local machine during the next step, it will execute alongside -with the cluster of machines you run in the cloud! That is, the Netherite load balancer will distribute -the partitions over all machines in the cloud, and your local machine. While this "super-cluster" would -function correctly, it is probably not what you want. +The setup of the Azure Functions app is language-specific. We provide a PowerShell script for the samples in the folder [/samples/scripts](https://github.com/microsoft/durabletask-netherite/tree/main/samples/scripts) namely: -## Deploy it to Azure +- C#: `create-function-app-dotnet.ps1` +- TypeScript: `create-function-app-node.ps1` +- Python: `create-function-app-python.ps1` -When running Netherite in the cloud, we use a *premium functions plan* for now, until we implement support for the consumption plan. +These scripts will: -First, build the release binaries: -```shell -dotnet build -c Release -``` +1. create a premium plan with the specified SKU +2. create a Function App +3. configure the Function App for Netherite -Then, if you want, review or change the following parameters in `deploy-to-premium.ps1`: +Before you execute the scripts, review and if applicable change the following parameters in the script (as default we chose the smallest node size _EP1_. For heavier loads EP2 or EP3 may be more appropriate.): -```PowerShell -$Plan="EP1", -$MinNodes="1", -$MaxNodes="20", -$Configuration="Release" +```powershell +param ( + $Plan = "EP1", + $MinNodes = "1", + $MaxNodes = "20", + ... +) ``` -We chose the smallest node size (EP1); for heavier loads EP2 or EP3 may be more appropriate. +After you have executed the script you can deploy the Function. + +### Deploy Azure Function to Azure -Finally, run the `deploy-to-premium.ps1` script. This can take some time. As it executes, the script +You can trigger the deployment of the Function via the corresponding PowerShell script in the language-specific sample folder: -1. Creates a premium plan with the specified SKU -2. Create a function app -2. Configures the app -3. Deploys the code to the app +- C#: `deploy-dotnet-function.ps1` +- TypeScript: `deploy-typescript-function.ps1` +- Python: `deploy-python-function.ps1` -After executing this script successfully, near the bottom of the output, you should see a list of all three published functions: +These scripts build and deploy you Function to the Function App created in the previous section. -```text +After the script has finished successfully, you should see a list of all three published functions near the bottom of the output including the `Invoke url` for the HTTP trigger. + +Here is an example output for .NET: + +```powershell HelloCities - [httpTrigger] Invoke url: https://.azurewebsites.net/api/hellocities @@ -153,23 +305,20 @@ After executing this script successfully, near the bottom of the output, you sho SayHello - [activityTrigger] ``` -You can now *test the app that is running the cloud* by issuing an HTTP request: - -```shell -curl https://.azurewebsites.net/api/hellocities -``` +📝 **Remark** You find the URL also in the Azure portal. For the Python-based deployment this is the preferred way, as the output will not show up. -Which produces the following output, as expected: -``` -["Tokyo","Seattle","London"] -``` +You can now *test the app that is running the cloud* by issuing an HTTP request to the corresponding invocation endpoint. This should produce the same output as in the case of the local or hybrid scenarios executed before. If you need to update the application, you can rebuild it and redeploy it the same way. You can also change MinNodes or MaxNodes. This allows you to manually scale to a desired number. -## Delete Azure Resources +## Cleanup Azure Resources Don't forget to delete the resources when you're done, as they will continue to accrue charges. -To delete all the Azure Resources we created earlier, just execute the script `delete.ps1` and make sure to confirm your intention by hitting `y`. +To delete all the Azure resources, execute the script `delete.ps1` and make sure to confirm your intention by hitting `y`. + +### Taskhub only + +If you just want to clear the taskhub, but not delete the resources, you can run the script `clear.ps`. It deletes the containers in the Azure Blob Storage. You should stop the function app before doing so. -**Taskhub only** If you just want to clear the taskhub, but not delete the resources, you can run the script `clear.ps`. It deletes the containers in the Azure Blob Storage. You should probably stop the function app before doing so. Also, note that after you do this, you cannot restart the app for some time, because Azure Storage blob containers are unavailable after deletion for some time, maybe 30-60 seconds. +Note that after you do this, you cannot restart the app for some time, because Azure Storage blob containers are unavailable after deletion for some time (~ 30-60 seconds).