Test procedure for CB-Dragonfly monitoring in CB-Tumblebug

[seokho-son]

1) CB-DF용 MCIS(VM) 생성

CB-MapUI 를 통해서 생성하는 경우

• 스크립트로 MCIS 생성할 필요 없음
• VM 추가시, RootDisk를 200GB 이상으로 선택 필요 (CB-Dragonfly의 일반적인 요구사항)

CB-TB 스크립트로 생성

• 명령어
  ./create-all.sh -n tb-df -f ../testSet-df.env

• 구성 상세 정보

[CB-Tumblebug is READY]
[CB-Spider is READY]
####################################################################
## Check test config file (-n deveoperPrefix -f ../testSetCustom.env -x numOfVMsInEachVMGroup)
####################################################################


[Configuration in  ../testSet-df.env &  ../conf.env  files]

1) System Endpoints
 - Tumblebug Server :  localhost:1323 
 - Spider Server :  localhost:1024 

2) Enabled Clouds and Regions
 - [1] Cloud : aws (enabled regions : 1)
   [1,1] Region : aws-ap-southeast-1 (Asia Pacific (Singapore)) 
    - VM SPEC : m4.4xlarge 
    - VM IMAGE : Ubuntu 18.04 (ami-061eb2b23f9f8839c) 

3) MCIS Configuration
 - NameSpace ID : ns01
 - MCIS ID : tb-df
 - Number of Total VMs : 1
   - [1] VMs(1) = aws(1) * Replica(1)


Confirm the above configuration. Do you want to proceed ? (y/n) : y

---

• 생성 결과

Table: All VMs in the MCIS : tb-df
  
VM-ID                 Status   PublicIP       PrivateIP      CloudType  CloudRegion     CreatedTime
-----                 ------   --------       ---------      ---------  -----------     -----------
aws-ap-southeast-1-0  Running  18.141.203.29  192.168.2.140  aws        ap-southeast-1  2021-11-12   17:08:38


[DATE: 12/11/2021 17:08:39] [ElapsedTime: 36s (0m:36s)] [Command: ./create-mcis-only.sh all 1 tb-df ../testSet-df.env]

2) CB-DF용 VM에 CB-DF 서비스 설치 및 실행 (DockerCompose를 이용: https://github.com/cloud-barista/cb-operator/blob/master/docker-compose-mode-files/docker-compose-df-only.yaml)

---

• 명령어
  ./deploy-dragonfly-docker.sh -n tb-df -f ../testSet-df.env

CB-MapUI로 생성한 MCIS (id: tb-df) 인 경우, -f ../testSet-df.env 는 필요 없음.

• CB-DF 실행 및 CB-TB 환경설정 자동 처리 결과

####################################################################
## Deploy CB-Dragonfly
####################################################################

{
  "resultArray": [
    {
      "mcisId": "tb-df",
      "vmId": "aws-ap-southeast-1-0",
      "vmIp": "18.141.203.29",
      "result": "go build -o operator\ncb-operator is ready. Access to it with ssh 18.141.203.29 cb-dragonfly is ready. Access to it with ssh 18.141.203.29 "
    }
  ]
}
MASTERIP: 18.141.203.29
MASTERVM: aws-ap-southeast-1-0

[Waiting for initialization of CB-Dragonfly. (20s)]
Dozing for 20 : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 (Finish dozing. Back to work)
Checking CB-Dragonfly Status. (upto 3s * 20 trials)
CB-Dragonfly Status: 000
[1 : NOT Healthy.]..Dozing for 3 : 1 2 3 (Finish dozing. Back to work)
CB-Dragonfly Status: 000
[2 : NOT Healthy.]..Dozing for 3 : 1 2 3 (Finish dozing. Back to work)
CB-Dragonfly Status: 000
...
[19 : NOT Healthy.]..Dozing for 3 : 1 2 3 (Finish dozing. Back to work)
CB-Dragonfly Status: 000
[20 : NOT Healthy.]..Dozing for 3 : 1 2 3 (Finish dozing. Back to work)
CB-Dragonfly Status: 000
CB-Dragonfly is NOT healthy. We need to check manually. (Note: CB-DF needs 2 vCPU and 4GiB Memory at least)
If CB-Dragonfly is not responding, We need to wait more.

[Update Tumblebug Environment for Dragonfly with following command]
-x DRAGONFLY_REST_URL -y http://18.141.203.29:9090/dragonfly

####################################################################
## Config: Create or Update (-x: Key, -y: Value)
####################################################################
{
  "id": "DRAGONFLY_REST_URL",
  "name": "DRAGONFLY_REST_URL",
  "value": "http://18.141.203.29:9090/dragonfly"
}


[You can test Dragonfly with following command]
 ../9.monitoring/install-agent.sh all 1 tb-df
 ../9.monitoring/get-monitoring-data.sh all 1 tb-df cpu

[You can check Dragonfly monitoring status with Chronograf]
 (In Chronograf, you need to create a Dashboard view by selecting metrics)
 http://18.141.203.29:8888/sources/0/dashboards

• CB-DF 엔드포인트: http://18.141.203.29:9090/dragonfly
• 크로노그래프: http://18.141.203.29:8888/sources/0/dashboards

---

3) 모니터링 자동 설치되는 MCIS 생성

• 명령어
  ./create-all.sh -n mcis01 -f ../testSet01.env

• 설정 상세 (4종 클라우드)

[CB-Tumblebug is READY]
[CB-Spider is READY]
####################################################################
## Check test config file (-n deveoperPrefix -f ../testSetCustom.env -x numOfVMsInEachVMGroup)
####################################################################


[Configuration in  ../testSet01.env &  ../conf.env  files]

1) System Endpoints
 - Tumblebug Server :  localhost:1323 
 - Spider Server :  localhost:1024 

2) Enabled Clouds and Regions
 - [1] Cloud : aws (enabled regions : 1)
   [1,1] Region : aws-ap-southeast-1 (Asia Pacific (Singapore)) 
    - VM SPEC : m4.4xlarge 
    - VM IMAGE : Ubuntu 18.04 (ami-061eb2b23f9f8839c) 

 - [2] Cloud : azure (enabled regions : 1)
   [2,1] Region : azure-westus (West US) 
    - VM SPEC : Standard_B1s 
    - VM IMAGE : Ubuntu 18.04 (Canonical:UbuntuServer:18.04-LTS:18.04.202106220) 

 - [3] Cloud : gcp (enabled regions : 1)
   [3,1] Region : gcp-asia-east1 (Changhua County Taiwan [zone:a]) 
    - VM SPEC : e2-small 
    - VM IMAGE : Ubuntu 18.04 (https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/images/ubuntu-minimal-1804-bionic-v20191024) 

 - [4] Cloud : alibaba (enabled regions : 1)
   [4,1] Region : alibaba-ap-northeast-1 (Japan (Tokyo)) 
    - VM SPEC : ecs.t5-lc1m2.large 
    - VM IMAGE : Ubuntu 18.04 (ubuntu_18_04_x64_20G_alibase_20210927.vhd) 

3) MCIS Configuration
 - NameSpace ID : ns01
 - MCIS ID : mcis01
 - Number of Total VMs : 4
   - [1] VMs(1) = aws(1) * Replica(1)
   - [2] VMs(1) = azure(1) * Replica(1)
   - [3] VMs(1) = gcp(1) * Replica(1)
   - [4] VMs(1) = alibaba(1) * Replica(1)


Confirm the above configuration. Do you want to proceed ? (y/n) : y

---

• 생성 결과 ("monAgentStatus": "installed")

####################################################################
## 8. VM: Status MCIS
####################################################################

mcis01
{
  "status": {
    "id": "mcis01",
    "name": "mcis01",
    "status": "Running-4(4/4)",
    "statusCount": {
      "countTotal": 4,
      "countCreating": 0,
      "countRunning": 4,
      "countFailed": 0,
      "countSuspended": 0,
      "countRebooting": 0,
      "countTerminated": 0,
      "countSuspending": 0,
      "countResuming": 0,
      "countTerminating": 0,
      "countUndefined": 0
    },
    "targetStatus": "None",
    "targetAction": "None",
    "installMonAgent": "yes",
    "masterVmId": "alibaba-ap-northeast-1-0",
    "masterIp": "8.211.143.228",
    "masterSSHPort": "22",
    "vm": [
      {
        "id": "aws-ap-southeast-1-0",
        "name": "aws-ap-southeast-1-0",
        "cspVmId": "ns01-mcis01-aws-ap-southeast-1-0",
        "status": "Running",
        "targetStatus": "None",
        "targetAction": "None",
        "nativeStatus": "Running",
        "monAgentStatus": "installed",
        "systemMessage": "",
        "createdTime": "2021-11-12 17:20:34",
        "publicIp": "54.179.224.117",
        "privateIp": "192.168.2.138",
        "sshPort": "22",
        "location": {
          "latitude": "1.3700",
          "longitude": "103.8000",
          "briefAddr": "Singapore",
          "cloudType": "aws",
          "nativeRegion": "ap-southeast-1"
        }
      },
      {
        "id": "alibaba-ap-northeast-1-0",
        "name": "alibaba-ap-northeast-1-0",
        "cspVmId": "ns01-mcis01-alibaba-ap-northeast-1-0",
        "status": "Running",
        "targetStatus": "None",
        "targetAction": "None",
        "nativeStatus": "Running",
        "monAgentStatus": "installed",
        "systemMessage": "",
        "createdTime": "2021-11-12 17:20:39",
        "publicIp": "8.211.143.228",
        "privateIp": "192.168.5.133",
        "sshPort": "22",
        "location": {
          "latitude": "34.6895",
          "longitude": "139.9917",
          "briefAddr": "Japan (Tokyo)",
          "cloudType": "alibaba",
          "nativeRegion": "ap-northeast-1"
        }
      },
      {
        "id": "azure-westus-0",
        "name": "azure-westus-0",
        "cspVmId": "ns01-mcis01-azure-westus-0",
        "status": "Running",
        "targetStatus": "None",
        "targetAction": "None",
        "nativeStatus": "Running",
        "monAgentStatus": "installed",
        "systemMessage": "",
        "createdTime": "2021-11-12 17:21:01",
        "publicIp": "40.112.134.245",
        "privateIp": "192.168.3.132",
        "sshPort": "22",
        "location": {
          "latitude": "37.7830",
          "longitude": "-122.4170",
          "briefAddr": "West US",
          "cloudType": "azure",
          "nativeRegion": "westus"
        }
      },
      {
        "id": "gcp-asia-east1-0",
        "name": "gcp-asia-east1-0",
        "cspVmId": "ns01-mcis01-gcp-asia-east1-0",
        "status": "Running",
        "targetStatus": "None",
        "targetAction": "None",
        "nativeStatus": "Running",
        "monAgentStatus": "installed",
        "systemMessage": "",
        "createdTime": "2021-11-12 17:20:47",
        "publicIp": "34.80.131.226",
        "privateIp": "192.168.4.130",
        "sshPort": "22",
        "location": {
          "latitude": "24.0756",
          "longitude": "120.5451",
          "briefAddr": "Changhua County Taiwan",
          "cloudType": "gcp",
          "nativeRegion": "asia-east1"
        }
      }
    ]
  }
}

Table: All VMs in the MCIS : mcis01
  
VM-ID                     Status   PublicIP        PrivateIP      CloudType  CloudRegion     CreatedTime
-----                     ------   --------        ---------      ---------  -----------     -----------
alibaba-ap-northeast-1-0  Running  8.211.143.228   192.168.5.133  alibaba    ap-northeast-1  2021-11-12   17:20:39
aws-ap-southeast-1-0      Running  54.179.224.117  192.168.2.138  aws        ap-southeast-1  2021-11-12   17:20:34
azure-westus-0            Running  40.112.134.245  192.168.3.132  azure      westus          2021-11-12   17:21:01
gcp-asia-east1-0          Running  34.80.131.226   192.168.4.130  gcp        asia-east1      2021-11-12   17:20:47


[DATE: 12/11/2021 17:23:55] [ElapsedTime: 239s (3m:59s)] [Command: ./create-mcis-only.sh all 1 mcis01 ../testSet01.env]

---

온드멘드 모니터링 요청 시도 1 (gcp-asia-east1-0 실패)

---

• 명령어

../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu

• 결과

####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
{
  "nsId": "ns01",
  "mcisId": "mcis01",
  "mcisMonitoring": [
    {
      "metric": "cpu",
      "vmId": "gcp-asia-east1-0",
      "value": "",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "aws-ap-southeast-1-0",
      "value": "0.2493765586089154",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "alibaba-ap-northeast-1-0",
      "value": "3.030303030311728",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "azure-westus-0",
      "value": "9.803921568515563",
      "err": ""
    }
  ]
}

---

• CB-DF 호출 형태

[Call CB-DF] URL: http://18.141.203.29:9090/dragonfly/ns/ns01/mcis/mcis01/vm/gcp-asia-east1-0/agent_ip/34.80.131.226/metric/cpu/ondemand-monitoring-info

[Call CB-DF] URL: http://18.141.203.29:9090/dragonfly/ns/ns01/mcis/mcis01/vm/azure-westus-0/agent_ip/40.112.134.245/metric/cpu/ondemand-monitoring-info

• 오류 메시지

[Call CB-DF Result (mcis01,gcp-asia-east1-0)] HTTP Status code: 500
[CLOUD-BARISTA].[ERROR]: 2021-11-12 17:26:21 monitoring.go:477, github.com/cloud-barista/cb-tumblebug/src/core/mcis.CallGetMonitoringAsync() - HTTP Status: not in 200-399 

CB-DF Result: 
[Call CB-DF Result (mcis01,aws-ap-southeast-1-0)] HTTP Status code: 200
CB-DF Result: 0.2493765586089154

[Call CB-DF Result (mcis01,alibaba-ap-northeast-1-0)] HTTP Status code: 200
CB-DF Result: 3.030303030311728

[Call CB-DF Result (mcis01,azure-westus-0)] HTTP Status code: 200
CB-DF Result: 9.803921568515563

---

온드멘드 모니터링 요청 시도 2 (azure-westus-0 실패)

son@son:~/go/src/github.com/cloud-barista/cb-tumblebug/src/testclient/scripts/sequentialFullTest$ ../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu

####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
{
  "nsId": "ns01",
  "mcisId": "mcis01",
  "mcisMonitoring": [
    {
      "metric": "cpu",
      "vmId": "azure-westus-0",
      "value": "",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "aws-ap-southeast-1-0",
      "value": "0.12500000000272848",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "gcp-asia-east1-0",
      "value": "4.000000000007731",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "alibaba-ap-northeast-1-0",
      "value": "1.999999999998181",
      "err": ""
    }
  ]
}

---

온드멘드 모니터링 요청 시도 3 (azure-westus-0 실패)

son@son:~/go/src/github.com/cloud-barista/cb-tumblebug/src/testclient/scripts/sequentialFullTest$ ../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu

####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
{
  "nsId": "ns01",
  "mcisId": "mcis01",
  "mcisMonitoring": [
    {
      "metric": "cpu",
      "vmId": "azure-westus-0",
      "value": "",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "aws-ap-southeast-1-0",
      "value": "0.2493765586089154",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "gcp-asia-east1-0",
      "value": "9.803921568618705",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "alibaba-ap-northeast-1-0",
      "value": "2.020202020200159",
      "err": ""
    }
  ]
}

크로노그래프 접속 (푸시 모니터링 조회되지 않음. DockerCompose 상의 구성 오류일 수 있음)

[seokho-son]

cb-operator/docker-compose-mode-files/conf/cb-dragonfly/config.yaml

에 로컬 IP 주소를 외부에서 접속 가능한 주소로 변경해줘야 함.

cb-tumblebug/scripts/setcbdf.sh

Line 42 in ec62982

sudo sed -i "s/127.0.0.1/`curl https://api.ipify.org`/g" $GOPATH/src/github.com/cloud-barista/cb-operator/docker-compose-mode-files/conf/cb-dragonfly/config.yaml

기존 CB-TB의 CB-DF 자동 배포 스크립트에
관련 코드가 있었으나, DB-DF 0.4.3에서 outdated 되어 있었음. (@jihoon-seo WIP 에 배포시에도 CB-DF 설정에 유의할 필요가 있겠습니다..^^)

현재 수정하여 테스트 완료.

푸시 모니터링 확인

크로노그래프 접속 및 CPU. MEM, DISK, NET GUI 확인 => 문제 없음

온디멘드 모니터링 확인

병렬로 10회 수행 => 오류 없음. (기존 발생하던 이슈도 EXTERNAL IP에 의한 것으로 보임)

../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu & ../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &../9.monitoring/get-monitoring-data.sh -n mcis01 -f ../testSet01.env -x cpu &
[1] 7795
[2] 7796
[3] 7797
[4] 7798
[5] 7799
[6] 7800
[7] 7801
[8] 7802
[9] 7803
[10] 7804

####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
####################################################################
## Get MCIS monitoring data (parameter: -x [cpu/memory/disk/network])
####################################################################
{
  "nsId": "ns01",
  "mcisId": "mcis01",
  "mcisMonitoring": [
    {
      "metric": "cpu",
      "vmId": "alibaba-ap-northeast-1-0",
      "value": "12.371134020609858",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "aws-ap-southeast-1-0",
      "value": "3.12891113892303",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "gcp-europe-west3-0",
      "value": "34.34343434344083",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "alibaba-ap-south-1-0",
      "value": "14.141414141404397",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "gcp-asia-east1-0",
      "value": "39.583333333343205",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "azure-eastus-0",
      "value": "67.99999999998363",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "aws-ca-central-1-0",
      "value": "36.00000000001273",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "azure-westus-0",
      "value": "55.99999999999454",
      "err": ""
    }
  ]
}

...

{
  "nsId": "ns01",
  "mcisId": "mcis01",
  "mcisMonitoring": [
    {
      "metric": "cpu",
      "vmId": "alibaba-ap-northeast-1-0",
      "value": "7.92079207920257",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "aws-ap-southeast-1-0",
      "value": "1.4981273408115499",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "alibaba-ap-south-1-0",
      "value": "8.080808080802512",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "gcp-asia-east1-0",
      "value": "20.83333333336097",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "azure-westus-0",
      "value": "19.999999999990905",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "gcp-europe-west3-0",
      "value": "18.55670103091478",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "aws-ca-central-1-0",
      "value": "16.000000000023647",
      "err": ""
    },
    {
      "metric": "cpu",
      "vmId": "azure-eastus-0",
      "value": "20.40816326529665",
      "err": ""
    }
  ]
}