Killercoda scenerios Apache-beam-analysis

KubeEdge v1.14.2/Apache-beam-analysis/Readme.md

@@ -0,0 +1,23 @@
+# Data Analytics with Apache Beam
+
+## Description
+
+![High level architecture](images/High_level_Arch.png "High Level Architecture")
+
+The main aim of analytics engine is to get data from mqtt broker in stream format and apply rules on incoming data in real time and produce alert/action on mqtt broker. Getting data through pipeline and applying analysis function is done by using Apache Beam.
+
+###  Apache Beam
+
+Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, we can build a program that defines the pipeline.
+
+
+#### Why use Apache Beam for analytics
+
+There are many frameworks like Hadoop, Spark, Flink, Google Cloud Dataflow etc for stream processing. But there was no unified API to binds all such frameworks and data sources. It was needed to abstract out the application logic from these Big Data frameworks. Apache Beam framework provides this abstraction between your application logic and big data ecosystem.
+- A generic dataflow-based model for building an abstract pipeline which could be run on any runtime like Flink/Samza etc.
+- The same pipeline code can be executed on cloud(eg. Huawei Cloud Stream based on Apache Flink) and on the edge with a custom backend which can efficiently schedule workloads in an edge cluster and perform distributed analytics.
+- Apache Beam integrates well with TensorFlow for machine learning which is a key use-case for edge.
+- Beam has support for most of the functions required for stream processing and analytics.
+
+
+

KubeEdge v1.14.2/Apache-beam-analysis/index.json

@@ -0,0 +1,30 @@
+{
+    "title": "KubeEdge Deployment",
+    "description": "Data Analytics with Apache Beam",
+    "details": {
+        "steps": [
+            {
+                "title": "Step 1/4",
+                "text": "step1.md"
+            },
+            {
+                "title": "Step 2/4",
+                "text": "step2.md"
+            },
+            {
+                "title": "Step 3/4",
+                "text": "step3.md"
+            },
+            {
+                "title": "Step 4/4",
+                "text": "step4.md"
+            }
+        ],
+        "intro": {
+            "text": "intro.md"
+        }
+    },
+    "backend": {
+        "imageid": "ubuntu"
+    }
+}

KubeEdge v1.14.2/Apache-beam-analysis/intro.md

@@ -0,0 +1,31 @@
+# Data Analytics with Apache Beam
+
+## Description
+
+![High level architecture](images/High_level_Arch.png "High Level Architecture")
+
+The main aim of analytics engine is to get data from mqtt broker in stream format and apply rules on incoming data in real time and produce alert/action on mqtt broker. Getting data through pipeline and applying analysis function is done by using Apache Beam.
+
+###  Apache Beam
+
+Apache Beam is an open source, unified model for defining both batch and streaming data-parallel processing pipelines. Using one of the open source Beam SDKs, we can build a program that defines the pipeline.
+
+
+#### Demo 1.1 [Real-time alert]:Read batch data from MQTT,filter and generate alerts
+- Basic mqtt read/write support in Apache Beam for batch data
+- Reads data from an mqtt topic
+- Create PCollection of read data and use it as the initial data for pipeline
+- Do a filtering over the data
+- Publish an alert on a topic if reading exceeds the value
+![Demo1.1](images/Demo1.1.png "Demo1.1:Read batch data from MQTT,filter and generate alerts")
+
+#### Demo 1.2 [Filter Streaming Data]: Reads streaming data from MQTT, filter at regular intervals
+- Read streaming data using MQTT
+- Do a filtering over the data at fixed time intervals
+![demo1.2](images/Demo1.2.png "Demo1.2:Reads streaming data from MQTT, filter at regular intervals")
+
+### Prerequisites
+- Golang(version: 1.14+)
+- KubeEdge(version: v1.5+)
+- Docker(version: 18.09-ce+)
+

KubeEdge v1.14.2/Apache-beam-analysis/step1.md

@@ -0,0 +1,14 @@
+# Deploy pipeline application
+
+### Prerequisites
+
+- Golang(version: 1.14+)
+- KubeEdge(version: v1.5+)
+- Docker(version: 18.09-ce+)
+
+#### For demo 1.1: Pull the docker image from dockerhub by using following command
+
+```
+sudo docker pull containerise/ke_apache_beam:ke_apache_analysis_v1.1
+```{{execute}} 
+

KubeEdge v1.14.2/Apache-beam-analysis/step2.md

@@ -0,0 +1,22 @@
+# Deploy pipeline application
+
+### Prerequisites
+
+- Golang(version: 1.14+)
+- KubeEdge(version: v1.5+)
+- Docker(version: 18.09-ce+)
+
+#### For demo 1.2: Pull the docker image from dockerhub by using following command
+
+```
+sudo docker pull containerise/ke_apache_beam:ke_apache_analysis_v1.2
+```{{execute}} 
+
+#### Run the command
+
+This will shows all images created. Check image named ke_apache_analysis_v1.1 or ke_apache_analysis_v1.2
+
+```
+docker images
+```{{execute}}  
+

KubeEdge v1.14.2/Apache-beam-analysis/step3.md

@@ -0,0 +1,20 @@
+## Setup the KubeEdge v.1.24.2 
+follow this link 
+
+```
+https://killercoda.com/sarthak-009/scenario/deployment
+```
+
+
+### Try out a application deployment by following below steps.
+
+```
+kubectl apply -f https://github.com/kubeedge/examples/blob/master/apache-beam-analysis/deployment.yaml
+```{{execute}} 
+
+### Then you can use below command to check if the application is normally running.
+
+```
+kubectl get pods
+```{{execute}} 
+

KubeEdge v1.14.2/Apache-beam-analysis/step4.md

@@ -0,0 +1,23 @@
+## Clone the repository
+
+```
+git clone https://github.com/kubeedge/examples.git
+```{{execute}} 
+Change the directory to apache-beam-analysis
+
+```
+cd examples/apache-beam-analysis
+```{{execute}} 
+
+### Add following vendor packages:
+
+```
+go get -u github.com/yosssi/gmq/mqtt
+go get -u github.com/yosssi/gmq/mqtt/client
+```{{execute}} 
+
+run:
+```
+go build testmachine.go
+./testmachine
+```{{execute}} 

KubeEdge v1.14.2/README.md

@@ -0,0 +1,9 @@
+# KubeEdge Killercoda-Scenerio 
+
+We have created a tutorial in the interactive learning platform Killercoda for KubeEdge deployment. This can give a hands-on experience of KubeEdge deployment. The tutorial is created on KubeEdge release v1.14.2
+
+![alt text](/Images/scenarios.png) 
+
+This is available on <https://killercoda.com/sarthak-009>
+
+Pls try it out!!

KubeEdge v1.14.2/deployment/finish.md

@@ -0,0 +1 @@
+## Congratulations on successful completion of KubeEdge Deployment Scenario !!!

KubeEdge v1.14.2/deployment/index.json

@@ -0,0 +1,53 @@
+{
+    "title": "KubeEdge Deployment",
+    "description": "Deploying KubeEdge",
+    "details": {
+        "steps": [
+            {
+                "title": "Step 1/9",
+                "text": "step1.md"
+            },
+            {
+                "title": "Step 2/9",
+                "text": "step2.md"
+            },
+            {
+                "title": "Step 3/9",
+                "text": "step3.md"
+            },
+            {
+                "title": "Step 4/9",
+                "text": "step4.md"
+            },
+            {
+                "title": "Step 5/9",
+                "text": "step5.md"
+            },
+            {
+                "title": "Step 6/9",
+                "text": "step6.md"
+            },
+            {
+                "title": "Step 7/9",
+                "text": "step7.md"
+            },
+            {
+                "title": "Step 8/9",
+                "text": "step8.md"
+            },
+            {
+                "title": "Step 9/9",
+                "text": "step9.md"
+            }
+        ],
+        "intro": {
+            "text": "intro.md"
+        },
+        "finish": {
+            "text": "finish.md"
+        }
+    },
+    "backend": {
+        "imageid": "ubuntu"
+    }
+}

KubeEdge v1.14.2/deployment/intro.md

@@ -0,0 +1,5 @@
+### Let's deploy KubeEdge in 10 mins
+
+<br>
+
+ KubeEdge is built upon Kubernetes and extends native containerized application orchestration and device management to hosts at the Edge. It consists of cloud part and edge part, and provides core infrastructure support for networking, application deployment and metadata synchronisation between cloud and edge

KubeEdge v1.14.2/deployment/step1.md

@@ -0,0 +1,21 @@
+# Install kind   
+<br> 
+Kind is a tool for running local Kubernetes clusters using Docker container “nodes”.
+
+Run the command below to intsall kind:
+```
+curl -Lo ./kind "https://github.com/kubernetes-sigs/kind/releases/download/v0.14.0/kind-$(uname)-amd64"  
+chmod +x ./kind  
+mv ./kind /usr/local/bin/kind
+```{{execute}}  
+
+<br>
+<br>
+
+In order to manage the cluster later using the CLI, install Kubectl：
+
+```
+curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.22.6/bin/linux/amd64/kubectl
+chmod +x ./kubectl
+sudo mv ./kubectl /usr/local/bin/kubectl
+```{{execute}}

KubeEdge v1.14.2/deployment/step2.md

@@ -0,0 +1,8 @@
+# Create cluster  
+
+Run the command below to one-click create a cluster using kind. 
+
+```
+sudo kind create cluster
+
+```{{execute}}

KubeEdge v1.14.2/deployment/step3.md

@@ -0,0 +1,14 @@
+# Setup keadm
+
+Keadm is used to install the cloud and edge components of KubeEdge.      
+
+Run the command below to one-click install keadm.      
+
+``` 
+wget https://github.com/kubeedge/kubeedge/releases/download/v1.14.2/keadm-v1.14.2-linux-amd64.tar.gz
+tar -zxvf keadm-v1.14.2-linux-amd64.tar.gz
+sudo cp keadm-v1.14.2-linux-amd64/keadm/keadm /usr/local/bin/keadm
+
+```{{execute}}
+
+

KubeEdge v1.14.2/deployment/step4.md

@@ -0,0 +1,18 @@
+# Deploy cloudcore (on Master Node)
+
+keadm init will install cloudcore, generate the certs and install the CRDs.
+--advertise-address (non-mandatory flag) is the address exposed by the cloud side (will be added to the SANs of the CloudCore certificate), the default value is the local IP.
+
+```
+sudo keadm deprecated init --advertise-address="CloudCore-IP" --kubeedge-version=1.14.2 --kube-config=/root/.kube/config
+
+```{{execute}} 
+
+## check if cloudcore running successfully:
+
+```
+ps -elf | grep cloudcore
+
+```{{execute}}
+
+**Now you can see KubeEdge cloudcore is running.**

KubeEdge v1.14.2/deployment/step5.md

@@ -0,0 +1,72 @@
+# Setup edgecore(on Edge Node)
+
+In Kubernetes 1.23 and earlier, you could use Docker Engine with Kubernetes, relying on a built-in component of Kubernetes named dockershim. The dockershim component was removed in the Kubernetes 1.24 release; however, a third-party replacement, cri-dockerd, is available. The cri-dockerd adapter lets you use Docker Engine through the Container Runtime Interface.
+
+### Setup cri-dockerd 
+
+``` 
+wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd-0.3.4.amd64.tgz
+tar -xvf cri-dockerd-0.3.4.amd64.tgz
+cd cri-dockerd/
+mkdir -p /usr/local/bin
+install -o root -g root -m 0755 ./cri-dockerd /usr/local/bin/cri-dockerd
+
+```{{execute}}
+
+### Add the files cri-docker.socker cri-docker.service
+
+```
+sudo tee /etc/systemd/system/cri-docker.service << EOF
+[Unit]
+Description=CRI Interface for Docker Application Container Engine
+Documentation=https://docs.mirantis.com
+After=network-online.target firewalld.service docker.service
+Wants=network-online.target
+Requires=cri-docker.socket
+[Service]
+Type=notify
+ExecStart=/usr/local/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=
+ExecReload=/bin/kill -s HUP $MAINPID
+TimeoutSec=0
+RestartSec=2
+Restart=always
+StartLimitBurst=3
+StartLimitInterval=60s
+LimitNOFILE=infinity
+LimitNPROC=infinity
+LimitCORE=infinity
+TasksMax=infinity
+Delegate=yes
+KillMode=process
+[Install]
+WantedBy=multi-user.target
+EOF
+
+sudo tee /etc/systemd/system/cri-docker.socket << EOF
+[Unit]
+Description=CRI Docker Socket for the API
+PartOf=cri-docker.service
+[Socket]
+ListenStream=%t/cri-dockerd.sock
+SocketMode=0660
+SocketUser=root
+SocketGroup=docker
+[Install]
+WantedBy=sockets.target
+EOF
+
+
+```{{execute}}
+
+### Daemon reload
+
+```
+systemctl daemon-reload
+systemctl enable cri-docker.service
+systemctl enable --now cri-docker.socket
+systemctl start cri-docker.service
+
+```{{execute}}
+
+
+

KubeEdge v1.14.2/deployment/step6.md

@@ -0,0 +1,11 @@
+## Installing CNI plugin
+
+```
+wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
+
+mkdir -p /opt/cni/bin
+
+tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz
+
+```{{execute}}
+