Merge pull request #1 from mowangdk/feature/init-node-resource-manager

Add node resource manager features

CHANGELOG.md

@@ -0,0 +1,11 @@
+# CHANGELOG
+
+---
+
+### v0.1.0
+
+### node-resource-manager
+
+- Add LVM resource into node resource manager
+- Add QuotaPath resource into node resource manager 
+- Add KMEM resource into node resource manager

CODE_OF_CONDUCT.md

@@ -0,0 +1,5 @@
+# OpenYurt  Community Code of Conduct
+
+Node-resource-manager follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/master/code-of-conduct.md).
+
+In cases of abusive, harassing, or any unacceptable behaviors, please don't hesitate to contact the project team at [email protected].

Makefile

@@ -0,0 +1,20 @@
+# Copyright 2021 The OpenYurt Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+.PHONY: build
+build:
+# options:
+# 	ARCH=amd64
+#	VERSION=v1.0
+	bash ./build/build.sh $(OPTS)

PROJECT

@@ -0,0 +1,3 @@
+version: "1"
+domain: openyurt.io
+repo: github.com/openyurtio/node-resource-manager

README.md

@@ -0,0 +1,49 @@
+# openyurtio/node-resource-manager
+
+English | [简体中文](./README.zh.md)
+
+Node-resource-manager manages local node resources of OpenYurt cluster in a unified manner. 
+
+It currently manages:
+- LVM built on top of block device or pmem device.
+- QuotaPath built on top of block device or pmem device.
+- Memory built on top of pmem device. 
+
+The majority function consists of:
+- Initialize local resources on edge node.
+- Update local resources on edge node.
+
+You can define the spec of local resources by simply modifying the pre-defined ConfigMap.
+
+Node-resource-manager has the following advantages in terms of compatibility and usability.
+- **Easily to use**. Initialization and modification of local resources are easily done by editing the ConfigMap.
+- **Easily to integrate**. Node-resource-manager can work together with csi driver, to perform local storage lifecycle management.
+- **Platform free**. Node-resource-manager can be running in any kubernetes clusters.
+
+## Architecture
+The component consists of two parts, the first part is the ConfigMap named node-resource-topo in kube-system namespace,
+and the second is the node-resource-manager DaemonSet deployed in kube-system namespace.
+Node-resource-manager on each node mounts and reads the node-resource-topo ConfigMap to managed local resources.
+<div align="center">
+  <img src="docs/images/node-resource-manager.png" width=70% title="node-resource-manager architecture">
+</div>
+
+## Getting started
+
+1. Create node-resource-topo ConfigMap in kube-system namespace. ConfigMap example is in [configmap.md](./docs/configmap.md).
+```
+kubectl apply -f deploy/configmap.yaml
+```
+
+2. Deploy node-resource-manager DaemonSet.
+```
+kubectl apply -f deploy/nrm.yaml
+```
+
+## Developer guide
+
+Please refer to [developer-guide.md](./docs/developer-guide.md) for developing and building the project.
+
+## Roadmap
+
+[2021 Roadmap](docs/roadmap.md)

README.zh.md

@@ -0,0 +1,53 @@
+# openyurtio/node-resource-manager
+
+[English](./README.md) | 简体中文
+
+node-resource-manager 是用于管理 OpenYurt 集群本地资源的组件，用户可以通过修改集群内 ConfigMap 的定义来动态配置集群内宿主机上的本地资源。
+
+管理的本地资源包括：
+- 基于块设备或者是持久化内存设备创建的 LVM
+- 基于块设备或者是持久化内存设备创建的 QuotaPath
+
+主要功能包括：
+- 初始化节点上本地资源
+- 更新节点上的本地资源
+
+主要优点：
+- **简单易用**。node-resource-manager 可以仅通过定义 ConfigMap 就完成对集群中的本地资源的初始化和更新 
+- **易于集成**。 node-resource-manager 可以与 csi 插件集成来完成 kubernetes 集群中的相关本地资源的生命周期管理
+- **与云平台无关**。 node-resource-manager 可以轻松部署在任何公共云 Kubernetes 服务中。
+
+## 架构
+
+该组件主要包含两个部分， 一个是定义在集群中 kube-system namespace 的 node-resource-topo ConfigMap,
+一个是部署在集群中 kube-system namespace 下面的 node-resource-manager Daemonset, 
+每个 Node 节点上的 node-resource-manager 通过挂载 node-resource-topo ConfigMap 的方式生产并管理用户定义的本地资源。
+<div align="center">
+  <img src="docs/images/node-resource-manager.png" width=70% title="node-resource-manager architecture">
+</div>
+
+## 开始使用
+
+1. 在 Kubernetes kube-system namespace 下定义 node-resource-topo ConfigMap, 该 ConfigMap 用于定义集群中需要自动生成并管理的节点本地资源. 关于如何创建一个ConfigMap，请参见 ConfigMap [定义](./docs/configmap.zh.md)
+
+```
+kubectl apply -f deploy/configmap.yaml
+```
+
+2. 在 Kubernetes 集群中创建 node-resource-manager Daemonset。
+```
+kubectl apply -f deploy/nrm.yaml
+```
+
+3. 检查在 ConfigMap 上定义的资源是否都已经在对应的节点上被正确的创建。
+
+
+4. 配合 [alibaba-local-csi-plugin](https://help.aliyun.com/document_detail/178472.html?spm=a2c4g.11186623.6.844.13a019caYIiivY) 插件在集群中动态创建本地资源 pvc/pv, 供 pod 进行挂载使用。
+
+## 开发指南
+
+请参考 [developer-guide.md](./docs/developer-guide.md) 进行本项目的开发和构建。
+
+## 发展规划
+
+[2021年 发展规划](docs/roadmap.md)

build/build.sh

@@ -0,0 +1,118 @@
+# Copyright 2021 The OpenYurt Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#!/usr/bin/env bash
+set -x
+
+# usage:
+#  ./build.sh ARCH=amd64 VERSION=v1.0
+
+NRM_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd -P)"
+NRM_BUILD_DIR=${NRM_ROOT}/build
+NRM_OUTPUT_DIR=${NRM_BUILD_DIR}/_output
+NRM_BUILD_IMAGE="golang:1.13.3-alpine"
+
+GIT_VERSION="v1.0"
+GIT_VERSION=(${VERSION:-${GIT_VERSION}})
+GIT_SHA=`git rev-parse --short HEAD || echo "HEAD"`
+GIT_BRANCH=`git rev-parse --abbrev-ref HEAD 2>/dev/null`
+BUILD_TIME=`date "+%Y-%m-%d-%H:%M:%S"`
+
+IMG_REPO="openyurt/node-resource-manager"
+IMG_VERSION=${GIT_VERSION}-${GIT_SHA}
+
+readonly -a SUPPORTED_ARCH=(
+    amd64
+    arm
+    arm64
+)
+
+readonly -a target_arch=(${ARCH:-${SUPPORTED_ARCH[@]}})
+
+function build_multi_arch_binaries() {
+    local docker_run_opts=(
+        "-i"
+        "--rm"
+        "--network host"
+        "-v ${NRM_ROOT}:/opt/src"
+        "--env CGO_ENABLED=0"
+        "--env GOOS=linux"
+        "--env GIT_VERSION=${GIT_VERSION}"
+        "--env GIT_SHA=${GIT_SHA}"
+        "--env GIT_BRANCH=${GIT_BRANCH}"
+        "--env BUILD_TIME=${BUILD_TIME}"
+    )
+    # use goproxy if build from inside mainland China
+    [[ $region == "cn" ]] && docker_run_opts+=("--env GOPROXY=https://goproxy.cn")
+
+    local docker_run_cmd=(
+        "/bin/sh"
+        "-xe"
+        "-c"
+    )
+
+    local sub_commands="sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories; \
+        apk --no-cache add bash; \
+        cd /opt/src; "
+    for arch in ${target_arch[@]}; do
+        sub_commands+="CGO_ENABLED=0 GOOS=linux GOARCH='${arch}' go build \
+  -ldflags '-X main._BRANCH_=${GIT_BRANCH} -X main._VERSION_=${IMG_VERSION} -X main._BUILDTIME_=${BUILD_TIME}' -o build/_output/nrm.${arch} main.go; "
+    done
+
+    docker run ${docker_run_opts[@]} ${NRM_BUILD_IMAGE} ${docker_run_cmd[@]} "${sub_commands}"
+}
+
+function build_images() {
+    for arch in ${target_arch[@]}; do
+        local docker_file_path=${NRM_BUILD_DIR}/Dockerfile.$arch
+        local docker_build_path=${NRM_BUILD_DIR}
+        local nrm_image=$IMG_REPO:${IMG_VERSION}.${arch}
+        local base_image
+        case $arch in
+            amd64)
+                base_image="amd64/alpine:3.10"
+                ;;
+            arm64)
+                base_image="arm64v8/alpine:3.10"
+                ;;
+            arm)
+                base_image="arm32v7/alpine:3.10"
+                ;;
+            *)
+                echo unknown arch $arch
+                exit 1
+        esac
+        cat << EOF > $docker_file_path
+FROM ${base_image}
+LABEL maintainers="OpenYurt Authors"
+LABEL description="OpenYurt Node Resource Manager"
+
+RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.aliyun.com/g' /etc/apk/repositories
+RUN apk update && apk add --no-cache ca-certificates file util-linux lvm2 xfsprogs e2fsprogs blkid
+COPY entrypoint.sh /entrypoint.sh
+COPY _output/nrm.${arch} /bin/nrm
+RUN chmod +x /bin/nrm && chmod +x /entrypoint.sh
+
+ENTRYPOINT ["/entrypoint.sh"]
+EOF
+        docker build --no-cache -t $nrm_image -f $docker_file_path $docker_build_path
+        docker save $nrm_image > ${NRM_OUTPUT_DIR}/node-resource-manager-${arch}.tar
+    done
+}
+
+rm -rf ${NRM_OUTPUT_DIR}
+mkdir -p ${NRM_OUTPUT_DIR}
+umask 0022
+build_multi_arch_binaries
+build_images

build/entrypoint.sh

@@ -0,0 +1,30 @@
+#!/bin/sh
+
+if uname -a | grep -i -q ubuntu; then
+  lvmLine=`/usr/bin/nsenter --mount=/proc/1/ns/mnt dpkg --get-selections lvm2 | grep install -w -i | wc -l`
+  if [ "$lvmLine" = "0" ]; then
+    /usr/bin/nsenter --mount=/proc/1/ns/mnt apt install lvm2 -y
+  fi
+else
+  lvmLine=`/usr/bin/nsenter --mount=/proc/1/ns/mnt rpm -qa lvm2 | wc -l`
+  if [ "$lvmLine" = "0" ]; then
+    /usr/bin/nsenter --mount=/proc/1/ns/mnt yum install lvm2 -y
+  fi
+fi
+
+if [ "$lvmLine" = "0" ]; then
+    /usr/bin/nsenter --mount=/proc/1/ns/mnt sed -i 's/udev_sync\ =\ 0/udev_sync\ =\ 1/g' /etc/lvm/lvm.conf
+    /usr/bin/nsenter --mount=/proc/1/ns/mnt sed -i 's/udev_rules\ =\ 0/udev_rules\ =\ 1/g' /etc/lvm/lvm.conf
+    /usr/bin/nsenter --mount=/proc/1/ns/mnt systemctl restart lvm2-lvmetad.service
+    echo "install lvm and starting..."
+else
+    udevLine=`/usr/bin/nsenter --mount=/proc/1/ns/mnt cat /etc/lvm/lvm.conf | grep "udev_sync = 0" | wc -l`
+    if [ "$udevLine" != "0" ]; then
+        /usr/bin/nsenter --mount=/proc/1/ns/mnt sed -i 's/udev_sync\ =\ 0/udev_sync\ =\ 1/g' /etc/lvm/lvm.conf
+        /usr/bin/nsenter --mount=/proc/1/ns/mnt sed -i 's/udev_rules\ =\ 0/udev_rules\ =\ 1/g' /etc/lvm/lvm.conf
+        /usr/bin/nsenter --mount=/proc/1/ns/mnt systemctl restart lvm2-lvmetad.service
+        echo "update lvm.conf file: udev_sync from 0 to 1, udev_rules from 0 to 1"
+    fi
+fi
+
+/bin/nrm $@

deploy/configmap.yaml

@@ -0,0 +1,61 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: node-resource-topo
+  namespace: kube-system
+data:
+  volumegroup: |-
+    volumegroup:
+    - name: volumegroup1
+      key: kubernetes.io/hostname
+      operator: In
+      value: cn-zhangjiakou.192.168.3.114
+      topology:
+        type: device
+        devices:
+        - /dev/vdb
+        - /dev/vdc
+
+    - name: volumegroup1
+      key: kubernetes.io/hostname
+      operator: In
+      value: cn-beijing.192.168.3.35
+      topology:
+        type: pmem
+        regions:
+        - region0
+
+  quotapath: |-
+    quotapath:
+    - name: /mnt/path1
+      key: kubernetes.io/hostname
+      operator: In
+      value: cn-beijing.192.168.3.35
+      topology:
+        type: device
+        options: prjquota
+        fstype: ext4
+        devices:
+        - /dev/vdb
+
+    - name: /mnt/path2
+      key: kubernetes.io/hostname
+      operator: In
+      value: cn-beijing.192.168.3.36
+      topology:
+        type: pmem
+        options: prjquota,shared
+        fstype: ext4
+        regions: 
+        - region0
+
+  memory: |-
+    memory:
+    - name: test1
+      key: kubernetes.io/hostname
+      operator: In
+      value: cn-beijing.192.168.3.37
+      topology:
+        type: pmem
+        regions: 
+        - region0