Skip to content

Latest commit

 

History

History
848 lines (604 loc) · 30.1 KB

README.md

File metadata and controls

848 lines (604 loc) · 30.1 KB
Raw
page_type languages products urlFragment name description
sample
azdeveloper
java
bicep
azure
azure-kubernetes-service
pubsub-dapr-aks-java
Java Pub/Sub Sample using Kubernetes and DAPR
Demonstrate a pub/sub messaging architecture using Dapr for a Java application running in a Kubernetes cluster. This sample project implements an event-driven communication between different components of the system using the pub/sub pattern.

Java Pub/Sub Sample using Kubernetes and DAPR

This sample is to demonstrate a pub/sub messaging architecture using Dapr for a Java application running in a Kubernetes cluster. This sample project implements an event-driven communication between different components of the system using the pub/sub pattern. By leveraging Dapr, this sample provides an asynchronous processing in a distributed system and application of side car pattern in a Kubernetes cluster. Whether you are new to event-driven architectures or looking to explore Dapr's capabilities for a Java application running in Kubernetes, this sample is a great place to start.

This sample doesn't implement a great deal of distributed systems concerns like Idempotency, Fault Tolerance, Retries etc. at the application level. Instead, it focuses on the application architecture and how Dapr can be used to implement a pub/sub messaging pattern in a Java application running in Kubernetes.

The sample is built using Azure Developer CLI conventions to run on Azure. Alternatively, you can run the project in your local environment using Kind and Docker.

How it works?

This sample implements a simple pub/sub workflow:

  1. Public API endpoint receives new money transfer request. [TRANSFER(Sender: A, Amount: 100, Receiver:B)]
  2. Request is published to pub/sub broker.
  3. Transfer workflow starts
    1. Fraud service checks the legitimacy of the operation and triggers [VALIDATED(Sender: A, Amount: 100, Receiver:B)]
    2. Account service checks if Sender has enough funds and triggers [APPROVED(Sender: A, Amount: 100, Receiver: B)]
  4. Public API can be used to check if there is a confirmation of the money transfer request.

Workflow

Services

The project contains the following services:

  • Public API - Public API endpoint that receives new money transfer requests, starts workflow and checks customer notifications.
  • Fraud Service - Fraud service that checks the legitimacy of the operation.
  • Account Service - Account service that checks if Sender has enough funds.

Prerequisites

Following technologies and CLIs are used for the development. Follow the links to install them:

Alternatively you can use DevContainers and VS Code for local development. Opening the project with VS Code will automatically install all the required tools and extensions using DevContainers.

Getting Started

We use Make to automate the build and deployment process. You can run the following command to see the available commands:

make help

The following commands are available:

help                 💬 This help message :)
all-azure            🏃‍♀️ Run all the things in Azure
start-local          🧹 Setup local Kind Cluster
deploy-local         🚀 Deploy application resources locally
run-local            💿 Run app locally
port-forward-local   ⏩ Forward the local port
test                 🧪 Run tests, used for local development
clean                🧹 Clean up local files
dapr-dashboard       🔬 Open the Dapr Dashboard
dapr-components      🏗️  List the Dapr Components
deploy-azure         🚀 Deploy application resources in Azure
test-azure           🧪 Run tests in Azure

Local Dev Environment Setup

Local environment is setup using Kind. Kind is a tool for running local Kubernetes clusters using Docker container "nodes". Kind was primarily designed for testing Kubernetes itself, but can be used for local development or CI.

1. Local Environment Setup

Running following command to setup your local development environment using Docker and Kind:

make start-local

This script runs the followings:

  1. Creates a local Docker registry called kind-registry running locally on port 9999.
  2. Creates a Kind cluster called azd-aks using config file from kind-cluster-config.yaml.
  3. Connects the registry to the cluster network if not already connected so deployments can access the local registry.
  4. Maps the local registry to the cluster
  5. Deploys Redis to the cluster using Helm chart to use for different Dapr components (pub/sub, state store, etc).
  6. Deploys Dapr on your local cluster.
  7. Deploys Pub/Sub Broker using Redis as the message broker using redis.yaml Dapr component.

Your local cluster will be laid out as follows:

Local

2. Dapr Dashboard & Components

This will open the Dapr dashboard in your default browser. This assumes you installed Dapr CLI in your local machine.

make dapr-dashboard

You can validate that the setup of the dashboard finished successfully by navigating to http://localhost:9000.

To verify the installation of pub/sub broker and other components:

make dapr-components

This will output something similar to the following:

NAMESPACE  NAME                   TYPE          VERSION  SCOPES  CREATED              AGE
default    money-transfer-pubsub  pubsub.redis  v1               2023-05-10 11:25.24  10m
default    money-transfer-state   state.redis   v1               2023-05-10 11:25.24  10m

3. Deploy Services to Cluster

By convention, every service under /src folder has 2 files:

  1. Dockerfile - Dockerfile for building the service image.
  2. local-deploy.sh - Script file to build, publish and deployment the latest code to local cluster as Docker image.

To deploy all services to the cluster, run the following command under scripts folder:

make deploy-local

The deployment script will do the following:

  1. Build and publish the Docker image for each service.
  2. Deploy the image to the local cluster.
  3. Create a Kubernetes service for each service.
  4. Register the service with Dapr.

The output of the local deployment of each service will be similar to following:

🤖  Starting local deployments...


🎖️  Deploying Public API Service


🛖  Releasing version: 2023.05.10.11.38.59


☢️  Attempting to delete existing deployment public-api-service

Error from server (NotFound): deployments.apps "public-api-service" not found

🏗️  Building docker image

[+] Building 0.0s (19/19) FINISHED
 => [internal] load 
 
 ............
 
 => exporting to image                                                                                                                0.0s
 => => exporting layers                                                                                                               0.0s
 => => writing image sha256:0ee9a3ad60209d914d772661a07d64b49c95780b4e43f58457ea86018632d8cd                                          0.0s
 => => naming to localhost:5001/public-api-service:2023.05.10.11.38.59                                                                0.0s

🚚  Pushing docker image to local registry

The push refers to repository [localhost:5001/public-api-service]
.......
0f706090ed95: Layer already exists
a8dd5239cafe: Layer already exists
2023.05.10.11.38.59: digest: sha256:562162f9bfac78b6f234c372748ef20ce2457288a7c625bcb50b2ba5ed751798 size: 1993

🚀  Deploying to cluster

service/public-api-service created
deployment.apps/public-api-service created

🎉  Deployment complete

You can check the deployment status of the services:

kubectl get pods -A

You should get an output similar to:

NAMESPACE            NAME                                            READY   STATUS    RESTARTS       AGE
dapr-system          dapr-dashboard-575df59d4c-mp262                 1/1     Running   0              172m
dapr-system          dapr-operator-676b7df68d-xwzjw                  1/1     Running   0              172m
dapr-system          dapr-placement-server-0                         1/1     Running   1 (171m ago)   172m
dapr-system          dapr-sentry-5f44fd7c9d-gjjcl                    1/1     Running   0              172m
dapr-system          dapr-sidecar-injector-c66df4c49-h645k           1/1     Running   0              172m
default              account-service-655448db67-vdc6n                2/2     Running   0              14m
default              custody-service-5b8656d84c-lgb8j                2/2     Running   0              13m
default              fraud-service-7dfcd56d86-hs7s6                  2/2     Running   0              15m
default              notification-service-7dfbb47b4f-sthln           2/2     Running   0              12m
default              public-api-service-5df9f84648-gb9t2             2/2     Running   0              15m
default              redis-master-0                                  1/1     Running   0              172m
default              redis-replicas-0                                1/1     Running   0              172m
default              redis-replicas-1                                1/1     Running   0              171m
default              redis-replicas-2                                1/1     Running   0              170m
kube-system          coredns-565d847f94-k5nc4                        1/1     Running   0              172m
kube-system          coredns-565d847f94-qx2tn                        1/1     Running   0              172m
kube-system          etcd-azd-aks-control-plane                      1/1     Running   0              172m
kube-system          kindnet-7rhdn                                   1/1     Running   0              172m
kube-system          kindnet-bbk74                                   1/1     Running   0              172m
kube-system          kindnet-nfgrd                                   1/1     Running   0              172m
kube-system          kube-apiserver-azd-aks-control-plane            1/1     Running   0              172m
kube-system          kube-controller-manager-azd-aks-control-plane   1/1     Running   0              172m
kube-system          kube-proxy-6s47w                                1/1     Running   0              172m
kube-system          kube-proxy-7mnzh                                1/1     Running   0              172m
kube-system          kube-proxy-kj8pn                                1/1     Running   0              172m
kube-system          kube-scheduler-azd-aks-control-plane            1/1     Running   0              172m
local-path-storage   local-path-provisioner-684f458cdd-wtmqh         1/1     Running   0              172m

4. Interacting with the Application

Public API is deployed as Loadbalancer service type. In local cluster setup, it will not able to get public IP to connect. Instead, you can access this service locally using the Kubectl proxy tool.

make port-forward-local

While this command is running, you can access the service at http://localhost:8080.

export PUBLIC_API_SERVICE=http://localhost:8080

First, you need to create a new account for a user:

curl -X POST \
  $PUBLIC_API_SERVICE/accounts \
  -H 'Content-Type: application/json' \
  -d '{
    "owner": "A",
    "amount": 100
}'

This should return the following response:

{
  "account":
  {
    "owner":"A",
    "amount":100.0
  },
  "message":"Account created for: Owner: A, Amount: 100.0"
}

An example request to start a new transfer workflow is:

curl -X POST \
  $PUBLIC_API_SERVICE/transfers \
  -H 'Content-Type: application/json' \
  -d '{
    "sender": "A",
    "receiver": "B",
    "amount": 100
}'

This should return the following response:

{
  "message":"Transfer Request Started: Sender: A, Receiver: B, Amount: 100.0",
  "status":"ACCEPTED",
  "transferId":"14efc"
}

You can query the status of a transfer:

curl -X GET \
  $PUBLIC_API_SERVICE/transfers/{transferId} \
  -H 'Content-Type: application/json'

This should return the following response:

{
  "sender":"A",
  "receiver":"B",
  "amount":100.0,
  "transferId":"14efc",
  "status":"COMPLETED"
}

5. Delete Local Cluster

If you'd like to create a clean state and start over, you can run the following command to delete the local cluster and all the resources associated with it.

make clean

Implementation Status

  • Public API endpoint receives new money transfer request. [TRANSFER(Sender: A, Amount: 100, Receiver:B)]
  • Request is published to Redis (pub/sub)
  • Deposit workflow starts
    • Fraud service checks the legitimacy of the operation and triggers [VALIDATED(Sender: A, Amount: 100, Receiver:B)]
    • Account service checks if Sender has enough funds and triggers [APPROVED(Sender: A, Amount: 100, Receiver: B)]
  • Public API can be used to check if there is a confirmation of the money transfer request.

How To

Following are practical operator guides for common tasks.

How to Operate Redis

To connect to Redis, you can use the following command:

kubectl exec -it redis-master-0 -- redis-cli

To get Redis password:

kubectl get secret --namespace default redis -o jsonpath="{.data.redis-password}" | base64 --decode

To AUTH with Redis instance for further operations

AUTH <password>

Get all keys in the state store:

keys *

Azure Deployment

It's possible to deploy this application through Azure Developer (azd) CLI. azd is responsible for resource provision and application deployment requiring minimal interaction with Azure Portal.

For more information of azd CLI and how to Make your project azd-compatible

Pre-requisites

The following are available in the devContainer environment:

  • Azure Developer CLI
  • Azure CLI with K8s-extension enabled

The following have to be registered in the azure subscription after signing in:

  • AKS-ExtensionManager and AKS-Dapr features
  • Kubernetes and ContainerService resource providers

First, make sure you have signed in to Azure CLI:

az login

Then, set the subscription you want to use:

az account set --subscription <subscription-id>

When your subscription is set, you can check the current registration status of the features and resource providers:

az feature list --query "[?name=='Microsoft.ContainerService/AKS-Dapr' || name=='Microsoft.ContainerService/AKS-ExtensionManager'].{Name:name,State:properties.state}" --output table

Then, register the features and resource providers:

# register AKS-ExtensionManager and AKS-Dapr resource provider
az feature register --namespace "Microsoft.ContainerService" --name "AKS-ExtensionManager"
az feature register --namespace "Microsoft.ContainerService" --name "AKS-Dapr"

# invoke the resource providers for Kubernetes and ContainerService to propagate the registration
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ContainerService

1. Initiate Environment

We are going to use pubsub-dapr-aks-java as the environment name for this sample. To initiate an azd environment, run the following command:

azd init --environment pubsub-dapr-aks-java

The above command also adds folders to your project to store the environment variables it creates along the deployment process

├── .azure                     [ For storing Azure configurations]
│   └── <your environment>     [ For storing all environment-related configurations]
│      ├── .env                [ Contains environment variables ]
│      └── config.json         [ Contains environment configuration ]

2. Resource Provision and Deploy

By just running azd up, azd will provision all the resources needed for this sample and deploy the application to the cluster. This command combines azd provision and azd deploy to achieve this.

When issuing the command below, you need to select an Azure Subscription and Location. Also while installing Dapr post-provision, you need to approve the installation of Dapr components interactively.

azd up

This will output something like this:

......

Packaging services (azd package)

  (✓) Done: Packaging service account-service
  - Image Hash: sha256:f03ab78c43152afac5263975208301b994f328195741c5ae3cba407854c7722a
  - Image Tag: java--pubsub-dapr-sample/account-service-java--pubsub-dapr-sample:azd-deploy-1684851734
  (✓) Done: Packaging service fraud-service
  - Image Hash: sha256:7aa3ed7fdd27394d7bb9990c363b477f2ab0e7a9487338cc7f45203d00c3df7b
  - Image Tag: java--pubsub-dapr-sample/fraud-service-java--pubsub-dapr-sample:azd-deploy-1684851737
  (✓) Done: Packaging service notification-service
  - Image Hash: sha256:303bb6b3638f512601d99abba03b14ce74041e945abe8c9ba5a09623e57f518c
  - Image Tag: java--pubsub-dapr-sample/notification-service-java--pubsub-dapr-sample:azd-deploy-1684851740
  (✓) Done: Packaging service public-api-service
  - Image Hash: sha256:b4dcc470a2482b80cc97f2c21a1dd704c8a357c10d31c7b7f48c26ba4a39ae4d
  - Image Tag: java--pubsub-dapr-sample/public-api-service-java--pubsub-dapr-sample:azd-deploy-1684851744

  ......

  ✓) Done: Resource group: rg-java--pubsub-dapr-sample
  (✓) Done: Key vault: kv-5jkc2ah3kbvqw
  (✓) Done: Container Registry: cr5jkc2ah3kbvqw
  (✓) Done: AKS Managed Cluster: aks-5jkc2ah3kbvqw
Executing postprovision hook => ./infra/dapr-install.sh
📀 - Post-Provision hook - Installing DAPR AKS...
Merged "aks-5jkc2ah3kbvqw" as current context in /Users/mahmutcanga/.kube/config

......

- Creating java--pubsub-dapr-sample namespace...
namespace/java--pubsub-dapr-sample created

🚀 Deploy Redis on AKS

serviceaccount/redis created
secret/redis created
configmap/redis-configuration created
configmap/redis-health created
configmap/redis-scripts created
service/redis-headless created
service/redis-master created
service/redis-replicas created
statefulset.apps/redis-master created
statefulset.apps/redis-replicas created

🚀 Deploy pub-sub broker component backed by Redis

component.dapr.io/money-transfer-pubsub created

🚀 Deploy state store component backed Redis

component.dapr.io/money-transfer-state created


Deploying services (azd deploy)

  (✓) Done: Deploying service account-service
  - Endpoint: http://None:80, (Service, Type: ClusterIP)

  (✓) Done: Deploying service fraud-service
  - Endpoint: http://None:80, (Service, Type: ClusterIP)

  (✓) Done: Deploying service notification-service
  - Endpoint: http://None:80, (Service, Type: ClusterIP)

  (✓) Done: Deploying service public-api-service
  - Endpoint: http://20.101.13.7, (Service, Type: LoadBalancer)


SUCCESS: Your application was provisioned and deployed to Azure in 16 minutes 19 seconds.

The Azure Developer CLI supports various extension points to customize your workflows and deployments. The hooks middleware allows you to execute custom scripts before and after azd commands and service lifecycle events. In our setup, we are using postprovision hook to install:

  • Dapr
  • Redis
  • Dapr Redis State Store
  • Dapr Redis Pub/Sub

3. Interact with the application

At this point, you have everything configured in your Azure environment. From here, you can start interacting with the application.

The environment variables created along the deployment process could be used to interact with the application. You can run the following command to set the environment variables:

source <(azd env get-values)

To check if the API is running and healthy, let's get the public api of the public-api-service by running the following command:

kubectl get service public-api-service -n $AZURE_ENV_NAME

This will output something like this:

NAME                 TYPE           CLUSTER-IP   EXTERNAL-IP    PORT(S)        AGE
public-api-service   LoadBalancer   10.0.54.16   20.8.255.153   80:31319/TCP   2m27s

You can use the EXTERNAL-IP to access the API. You can run the following command to set the API endpoint available as environment variable:

PUBLIC_API_SERVICE=$(k get svc public-api-service -n $AZURE_ENV_NAME -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')

To check if the API is running and healthy, run the following command:

curl $PUBLIC_API_SERVICE

This will output something like this:

{"message":"Public API Service Started, version: 1.0.0"}

4. Test the application

First, you need to create a new account for a user:

curl -X POST \
  $PUBLIC_API_SERVICE/accounts \
  -H 'Content-Type: application/json' \
  -d '{
    "owner": "A",
    "amount": 100
}'

This should return the following response:

{
  "account":
  {
    "owner":"A",
    "amount":100.0
  },
  "message":"Account created for: Owner: A, Amount: 100.0"
}

An example request to start a new transfer workflow is:

curl -X POST \
  $PUBLIC_API_SERVICE/transfers \
  -H 'Content-Type: application/json' \
  -d '{
    "sender": "A",
    "receiver": "B",
    "amount": 100
}'

This should return the following response:

{
  "message":"Transfer Request Started: Sender: A, Receiver: B, Amount: 100.0",
  "status":"ACCEPTED",
  "transferId":"14efc"
}

You can query the status of a transfer:

curl -X GET \
  $PUBLIC_API_SERVICE/transfers/{transferId} \
  -H 'Content-Type: application/json'

This should return the following response:

{
  "sender":"A",
  "receiver":"B",
  "amount":100.0,
  "transferId":"14efc",
  "status":"COMPLETED"
}

5. Clean up

This will delete all the resources created by azd, including the resource group created for this sample by running the following command:

azd down --force --purge

This will output something like this:

Deleting all resources and deployed code on Azure (azd down)
Local application code is not deleted when running 'azd down'.

Deleting your resources can take some time.

  (✓) Done: Deleting resource group: rg-pubsub-dapr-sample-dev

  (✓) Done: Purging key vault: kv-wrxjknyqx2vrk

SUCCESS: Your application was removed from Azure in 11 minutes 13 seconds.

6. Manual Provisioning

Alternatively, you can provision and deploy the application manually without the hook if you'd require further customization in your workflow.

To deploy all the resources needed for this sample, run the following

azd provision

This will set up a dedicated resource group in your subscription with a clean AKS cluster, a Container Registry for the cluster to pull images from, and a Keyvault for secrets' storage (needed for the AKS deployed through azd).

6.1. Install Dapr and Redis on the Cluster

The variables used on the following commands are generated by azd and stored in .azure/<your_env_name>/.env

To use these environment variables, you can run the following command:

source <(azd env get-values)
# Install DAPR on AKS Cluster
az k8s-extension create --cluster-type managedClusters \
  --cluster-name $AZURE_AKS_CLUSTER_NAME \
  --resource-group $AZURE_RESOURCE_GROUP_NAME \
  --name myDaprExtension \
  --extension-type Microsoft.Dapr

To connect to the cluster and run other steps, update your current context of kubectl by running the following command:

az aks get-credentials --resource-group $AZURE_RESOURCE_GROUP_NAME --name $AZURE_AKS_CLUSTER_NAME

To check if Dapr is installed, run the following command:

kubectl get pods -n dapr-system

This will output something like this:

NAME                                    READY   STATUS    RESTARTS   AGE
dapr-dashboard-7798c78c74-4ddw5         1/1     Running   0          5m9s
dapr-operator-8955455bf-6qs79           1/1     Running   0          5m9s
dapr-placement-server-0                 1/1     Running   0          5m9s
dapr-sentry-69c8464dc9-6rtkf            1/1     Running   0          5m9s
dapr-sidecar-injector-fcbc4b979-wcjmj   1/1     Running   0          5m9s

We will deploy the application to its own namespace. Run following command to create a namespace for the application:

kubectl create namespace $AZURE_ENV_NAME

Check if the namespace is created:

kubectl get namespace

This will output something like this:

NAME                                  STATUS   AGE
dapr-system                           Active   6m48s
default                               Active   122m
gatekeeper-system                     Active   121m
java-pubsub-dapr-sample-dev           Active   25s   // <--- This is the namespace we created using $AZURE_ENV_NAME
kube-node-lease                       Active   122m
kube-public                           Active   122m
kube-system                           Active   122m

We are using an in-cluster Redis instance for the pub-sub and state store components. To deploy Redis, run the following command:

kubectl apply -f ./infra/redis.yaml --namespace $AZURE_ENV_NAME --wait=true

You can check the Redis installation by running the following command:

kubectl get pods --namespace $AZURE_ENV_NAME

This will output something like this:

NAME               READY   STATUS    RESTARTS      AGE
redis-master-0     1/1     Running   0             2m50s
redis-replicas-0   0/1     Running   3 (28s ago)   2m50s

6.2. Deploy PubSub and State Store components

Now, we have an in-cluster Redis instance running. We can deploy the pub-sub and state store components for Dapr.

6.2.1. Deploy pub-sub broker component backed by Redis
kubectl apply -f ./local/components/pubsub.yaml --wait=true --namespace $AZURE_ENV_NAME

You can check the pub-sub broker installation by running the following command:

dapr components -k --namespace $AZURE_ENV_NAME

This will output something like this:

NAMESPACE                            NAME                   TYPE          VERSION  SCOPES  CREATED              AGE
java--pubsub-dapr-sample-dev  money-transfer-pubsub  pubsub.redis  v1               2023-05-22 16:55.50  56s
6.2.2. Deploy state store component backed Redis
kubectl apply -f ./local/components/state.yaml --wait=true --namespace $AZURE_ENV_NAME

You can check the state store component installation by running the following command:

dapr components -k --namespace $AZURE_ENV_NAME

This will output something like this:

NAMESPACE                            NAME                   TYPE          VERSION  SCOPES  CREATED              AGE
java--pubsub-dapr-sample-dev  money-transfer-pubsub  pubsub.redis  v1               2023-05-22 16:55.50  2m
java--pubsub-dapr-sample-dev  money-transfer-state   state.redis   v1               2023-05-22 16:57.46  13s
6.2.3. Deploy microservices

Previously, we have setup our cluster, deployed Dapr and its components. Now, we can deploy the microservices.

Before you start the deployment, you will need to login to Azure Container Registry (ACR) to be able to push the images. To do so, run the following command. This command logs in to an Azure Container Registry through the Docker CLI. Docker must be installed on your machine. Once done, use 'docker logout ' to log out. (If you only need an access token and do not want to install Docker, specify '--expose-token').

az acr login --name $AZURE_CONTAINER_REGISTRY_NAME

This will output something like this:

Login Succeeded

This command deploys all services defined in azure.yaml file in the namespace defined by the environment name, based on their individual deployment under manifests folder by azd convention.

azd deploy

This will output something like this:

Deploying services (azd deploy)

  (✓) Done: Deploying service account-service
  - Endpoint: http://None:80, (Service, Type: ClusterIP)

  (✓) Done: Deploying service fraud-service
  - Endpoint: http://None:80, (Service, Type: ClusterIP)

  (✓) Done: Deploying service notification-service
  - Endpoint: http://None:80, (Service, Type: ClusterIP)

  (✓) Done: Deploying service public-api-service
  - Endpoint: http://20.229.248.100, (Service, Type: LoadBalancer)


SUCCESS: Your application was deployed to Azure in 12 minutes 13 seconds.
You can view the resources created under the resource group rg-java--pubsub-dapr-sample-dev in Azure Portal:
https://portal.azure.com/#@/resource/subscriptions/a3ed6c04-563f-4855-ac84-bdf1e5fbc3fc/resourceGroups/rg-java--pubsub-dapr-sample-dev/overview