Add Statefulsets #29

Aergonus · 2018-01-03T00:04:46Z

Verified working in 1.8 also has namespaces but it doesn't affect this test

-sh-4.2$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-08T18:39:33Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-08T18:27:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Builds upon #28

Test SS yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: test-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: counter-ss
  name: counter-ss
spec:
  clusterIP: None
  ports:
  - port: 9001
  selector:
    app: counter-ss
---
  apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: counter-ss
    namespace: test-system
  spec:
    serviceName: counter-ss
    replicas: 3
    template:
      metadata:
        labels:
          kube-monkey/enabled: enabled
          kube-monkey/identifier: monkey-victim-counter-ss
          kube-monkey/mtbf: "1"
          kube-monkey/kill-all: "kill-all"
      spec:
        containers:
        - args:
          - /bin/sh
          - -c
          - 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'
          image: testbox:latest
          imagePullPolicy: Always
          name: test-counter-ss
        restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: counter-delete-all-fail-test-ss
  name: counter-delete-all-fail-test-ss
spec:
  clusterIP: None
  ports:
  - port: 9002
  selector:
    app: counter-delete-all-fail-test-ss
---
  apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: counter-delete-all-fail-test-ss
    namespace: test-system
  spec:
    serviceName: counter-delete-all-fail-test-ss
    replicas: 3
    template:
      metadata:
        labels:
          kube-monkey/enabled: enabled
          kube-monkey/identifier: monkey-victim-delete-all-fail-test-ss
          kube-monkey/mtbf: "1"
          kube-monkey/kill-all: "kill-all"
      spec:
        containers:
        - args:
          - /bin/sh
          - -c
          - 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'
          image: DOESNOTEXIST:latest
          imagePullPolicy: Always
          name: test-counter-delete-all-fail-test-ss
        restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: singlecounter-ss
  name: singlecounter-ss
spec:
  clusterIP: None
  ports:
  - port: 9003
  selector:
    app: singlecounter-ss
---
  apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: singlecounter-ss
    namespace: test-system
  spec:
    serviceName: singlecounter-ss
    replicas: 1
    template:
      metadata:
        labels:
          kube-monkey/enabled: enabled
          kube-monkey/identifier: monkey-victim-single-counter-ss
          kube-monkey/mtbf: "2"
      spec:
        containers:
        - args:
          - /bin/sh
          - -c
          - 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'
          image: testbox:latest
          imagePullPolicy: Always
          name: test-single-counter-ss
        restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: opt-out-counter-ss
  name: opt-out-counter-ss
spec:
  clusterIP: None
  ports:
  - port: 9004
  selector:
    app: opt-out-counter-ss
---
  apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: opt-out-counter-ss
    namespace: test-system
  spec:
    serviceName: opt-out-counter-ss
    replicas: 1
    template:
      metadata:
        labels:
          kube-monkey/enabled: disabled
          kube-monkey/identifier: not-a-counter-victim-ss
          kube-monkey/mtbf: "1"
      spec:
        containers:
        - args:
          - /bin/sh
          - -c
          - 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 86400; done'
          image: testbox:latest
          imagePullPolicy: Always
          name: not-a-counter-victim-ss
        restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: fail-deploy-ss
  name: fail-deploy-ss
spec:
  clusterIP: None
  ports:
  - port: 9005
  selector:
    app: fail-deploy-ss
---
  apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: fail-deploy-ss
    namespace: test-system
  spec:
    serviceName: fail-deploy-ss
    replicas: 2
    template:
      metadata:
        labels:
          kube-monkey/enabled: enabled
          kube-monkey/identifier: failure-test-ss
          kube-monkey/mtbf: "1"
      spec:
        containers:
        - args:
          - /bin/sh
          - -c
          - 'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 300; done'
          image: invalidimage
          imagePullPolicy: Always
          name: fail-deploy-ss
        restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: secret-pod-ss
  name: secret-pod-ss
spec:
  clusterIP: None
  ports:
  - port: 9006
  selector:
    app: secret-pod-ss
---
# missing-secret.yaml
# a failed deployment
  apiVersion: apps/v1beta1
  kind: StatefulSet
  metadata:
    name: secret-pod-ss
    namespace: test-system
  spec:
    serviceName: secret-pod-ss
    replicas: 2
    template:
      metadata:
        labels:
          kube-monkey/enabled: enabled
          kube-monkey/identifier: missing-secret-test-ss
          kube-monkey/mtbf: "1"
      spec:
        containers:
          - name: missing-secret-test-ss
            image: testbox:latest
            command: [ "/bin/sh", "-c", "env" ]
            volumeMounts:
              - mountPath: /etc/secret/
                name: myothersecret
        volumes:
          - name: myothersecret
            secret:
              secretName: myothersecret
        restartPolicy: Always

Sample output:

Every 2.0s: kubectl logs kube-monkey-7bccdc67d6-6dwbt --namespace=kube-system                                                                                           Tue Jan  2 23:59:21 2018│Every 2.0s: kubectl get pods --namespace=test-system                                                Tue Jan  2 23:59:21 2018
                                                                                                                                                                                                │
I0102 23:57:17.485340       1 config.go:71] Successfully validated configs                                                                                                                      │NAME                                READY     STATUS              RESTARTS   AGE
I0102 23:57:17.486149       1 main.go:52] Starting kube-monkey with v logging level 4 and local log directory /var/log/kube-monkey                                                              │counter-delete-all-fail-test-ss-0   0/1       InvalidImageName    0          5s
E0102 23:57:17.486189       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│counter-ss-0                        0/1       Terminating         0          50s
account/ca.crt: no such file or directory                                                                                                                                                       │counter-ss-1                        0/1       Terminating         0          29s
I0102 23:57:17.490297       1 kubemonkey.go:19] Debug mode detected!                                                                                                                            │counter-ss-2                        0/1       Terminating         0          27s
I0102 23:57:17.490305       1 kubemonkey.go:20] Status Update: Generating next schedule in 10 sec                                                                                               │fail-deploy-ss-0                    0/1       ImagePullBackOff    0          28m
I0102 23:57:27.490452       1 schedule.go:54] Status Update: Generating schedule for terminations                                                                                               │opt-out-counter-ss-0                1/1       Running             0          28m
E0102 23:57:27.490549       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│secret-pod-ss-0                     0/1       ContainerCreating   0          28m
account/ca.crt: no such file or directory                                                                                                                                                       │singlecounter-ss-0                  1/1       Running             0          1m
I0102 23:57:27.542668       1 schedule.go:47] Status Update: 5 terminations scheduled today                                                                                                     │
I0102 23:57:27.542688       1 schedule.go:49] *v1beta1.StatefulSet counter-delete-all-fail-test-ss scheduled for termination at 01/02/2018 18:57:34 -0500 EST                                   │
        ********** Today's schedule **********                                                                                                                                                  │
I0102 23:57:27.542692       1 schedule.go:49] *v1beta1.StatefulSet counter-ss scheduled for termination at 01/02/2018 18:58:10 -0500 EST                                                        ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
        k8 Api Kind     Kind Name               Termination Time                                                                                                                                │# Please edit the object below. Lines beginning with a '#' will be ignored,
I0102 23:57:27.542695       1 schedule.go:49] *v1beta1.StatefulSet fail-deploy-ss scheduled for termination at 01/02/2018 18:58:23 -0500 EST                                                    │# and an empty file will abort the edit. If an error occurs while saving this file will be
        -----------     ---------               ----------------                                                                                                                                │# reopened with the relevant failures.
        *v1beta1.StatefulSet    counter-delete-all-fail-test-ss         01/02/2018 18:57:34 -0500 EST                                                                                           │#
        *v1beta1.StatefulSet    counter-ss              01/02/2018 18:58:10 -0500 EST                                                                                                           │apiVersion: v1
        *v1beta1.StatefulSet    fail-deploy-ss          01/02/2018 18:58:23 -0500 EST                                                                                                           │data:
        *v1beta1.StatefulSet    secret-pod-ss           01/02/2018 18:57:30 -0500 EST                                                                                                           │  config.toml: |
        *v1beta1.StatefulSet    singlecounter-ss                01/02/2018 18:57:34 -0500 EST                                                                                                   │    [kubemonkey]
        ********** End of schedule **********                                                                                                                                                   │    dry_run = false
I0102 23:57:27.542698       1 schedule.go:49] *v1beta1.StatefulSet secret-pod-ss scheduled for termination at 01/02/2018 18:57:30 -0500 EST                                                     │    time_zone = "America/New_York"
I0102 23:57:27.542701       1 schedule.go:49] *v1beta1.StatefulSet singlecounter-ss scheduled for termination at 01/02/2018 18:57:34 -0500 EST                                                  │    run_hour = 8
I0102 23:57:27.542778       1 kubemonkey.go:63] Status Update: Waiting to run scheduled terminations.                                                                                           │    start_hour = 10
E0102 23:57:30.542892       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│    end_hour = 16
account/ca.crt: no such file or directory                                                                                                                                                       │    graceperiod_sec = 10
I0102 23:57:30.555820       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet secret-pod-ss                                                                        │    blacklisted_namespaces = ["default","kube-system"]
I0102 23:57:30.555835       1 kubemonkey.go:74] Status Update: 4 scheduled terminations left.                                                                                                   │    whitelisted_namespaces = [""]
E0102 23:57:34.542756       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│    [debug]
account/ca.crt: no such file or directory                                                                                                                                                       │    enabled = true
E0102 23:57:34.542904       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│    schedule_delay = 10
account/ca.crt: no such file or directory                                                                                                                                                       │    force_should_kill = false
I0102 23:57:34.548276       1 chaos.go:125] Terminating ALL pods for *v1beta1.StatefulSet counter-delete-all-fail-test-ss                                                                       │    schedule_immediate_kill = true
E0102 23:57:34.551168       1 victims.go:140] Terminating pod singlecounter-ss-0 for *v1beta1.StatefulSet singlecounter-ss                                                                      │    [kubernetes]
I0102 23:57:34.554408       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet singlecounter-ss                                                                     │    host = "http://tlmtry-kubernetes-stage-ny.bdns.bloomberg.com:8080"
I0102 23:57:34.554430       1 kubemonkey.go:74] Status Update: 3 scheduled terminations left.                                                                                                   │kind: ConfigMap
I0102 23:57:34.554854       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet counter-delete-all-fail-test-ss                                                      │metadata:
I0102 23:57:34.554864       1 kubemonkey.go:74] Status Update: 2 scheduled terminations left.                                                                                                   │  creationTimestamp: 2018-01-02T23:53:01Z
E0102 23:58:10.542821       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│  name: kube-monkey-config-map
account/ca.crt: no such file or directory                                                                                                                                                       │  namespace: kube-system
I0102 23:58:10.546677       1 chaos.go:125] Terminating ALL pods for *v1beta1.StatefulSet counter-ss                                                                                            │  resourceVersion: "4346388"
I0102 23:58:10.557688       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet counter-ss                                                                           │  selfLink: /api/v1/namespaces/kube-system/configmaps/kube-monkey-config-map
I0102 23:58:10.557700       1 kubemonkey.go:74] Status Update: 1 scheduled terminations left.                                                                                                   │  uid: 10fddf9c-f018-11e7-a83c-a0481cb92994
E0102 23:58:23.542762       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│
account/ca.crt: no such file or directory                                                                                                                                                       ├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
I0102 23:58:23.548883       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet fail-deploy-ss                                                                       │---
I0102 23:58:23.548893       1 kubemonkey.go:74] Status Update: 0 scheduled terminations left.                                                                                                   │apiVersion: v1
I0102 23:58:23.548899       1 kubemonkey.go:77] Status Update: All terminations done.                                                                                                           │kind: Namespace
I0102 23:58:23.548945       1 kubemonkey.go:19] Debug mode detected!                                                                                                                            │metadata:
I0102 23:58:23.548960       1 kubemonkey.go:20] Status Update: Generating next schedule in 10 sec                                                                                               │  name: test-system
I0102 23:58:33.549143       1 schedule.go:54] Status Update: Generating schedule for terminations                                                                                               │---
E0102 23:58:33.549260       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│apiVersion: v1
account/ca.crt: no such file or directory                                                                                                                                                       │kind: Service
I0102 23:58:33.560884       1 schedule.go:47] Status Update: 4 terminations scheduled today                                                                                                     │metadata:
I0102 23:58:33.560899       1 schedule.go:49] *v1beta1.StatefulSet counter-delete-all-fail-test-ss scheduled for termination at 01/02/2018 18:59:14 -0500 EST                                   │  labels:
I0102 23:58:33.560903       1 schedule.go:49] *v1beta1.StatefulSet counter-ss scheduled for termination at 01/02/2018 18:59:10 -0500 EST                                                        │    app: counter-ss
I0102 23:58:33.560906       1 schedule.go:49] *v1beta1.StatefulSet fail-deploy-ss scheduled for termination at 01/02/2018 18:59:15 -0500 EST                                                    │  name: counter-ss
I0102 23:58:33.560910       1 schedule.go:49] *v1beta1.StatefulSet secret-pod-ss scheduled for termination at 01/02/2018 18:58:48 -0500 EST                                                     │spec:
I0102 23:58:33.560942       1 kubemonkey.go:63] Status Update: Waiting to run scheduled terminations.                                                                                           │  clusterIP: None
        ********** Today's schedule **********                                                                                                                                                  │  ports:
        k8 Api Kind     Kind Name               Termination Time                                                                                                                                │  - port: 9001
        -----------     ---------               ----------------                                                                                                                                │  selector:
        *v1beta1.StatefulSet    counter-delete-all-fail-test-ss         01/02/2018 18:59:14 -0500 EST                                                                                           │    app: counter-ss
        *v1beta1.StatefulSet    counter-ss              01/02/2018 18:59:10 -0500 EST                                                                                                           │---
        *v1beta1.StatefulSet    fail-deploy-ss          01/02/2018 18:59:15 -0500 EST                                                                                                           │  apiVersion: apps/v1beta1
        *v1beta1.StatefulSet    secret-pod-ss           01/02/2018 18:58:48 -0500 EST                                                                                                           │  kind: StatefulSet
        ********** End of schedule **********                                                                                                                                                   │  metadata:
E0102 23:58:48.561101       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│    name: counter-ss
account/ca.crt: no such file or directory                                                                                                                                                       │    namespace: test-system
I0102 23:58:48.567234       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet secret-pod-ss                                                                        │  spec:
I0102 23:58:48.567245       1 kubemonkey.go:74] Status Update: 3 scheduled terminations left.                                                                                                   │    serviceName: counter-ss
E0102 23:59:10.560991       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│    replicas: 3
account/ca.crt: no such file or directory                                                                                                                                                       │    template:
I0102 23:59:10.564679       1 chaos.go:125] Terminating ALL pods for *v1beta1.StatefulSet counter-ss                                                                                            │      metadata:
I0102 23:59:10.575079       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet counter-ss                                                                           │        labels:
I0102 23:59:10.575090       1 kubemonkey.go:74] Status Update: 2 scheduled terminations left.                                                                                                   │          kube-monkey/enabled: enabled
E0102 23:59:14.560938       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│          kube-monkey/identifier: monkey-victim-counter-ss
account/ca.crt: no such file or directory                                                                                                                                                       │          kube-monkey/mtbf: "1"
I0102 23:59:14.564991       1 chaos.go:125] Terminating ALL pods for *v1beta1.StatefulSet counter-delete-all-fail-test-ss                                                                       │          kube-monkey/kill-all: "kill-all"
I0102 23:59:14.570821       1 kubemonkey.go:71] Termination successfully executed for *v1beta1.StatefulSet counter-delete-all-fail-test-ss                                                      │      spec:
I0102 23:59:14.570836       1 kubemonkey.go:74] Status Update: 1 scheduled terminations left.                                                                                                   │        containers:
E0102 23:59:15.561094       1 config.go:330] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: open /var/run/secrets/kubernetes.io/service│

Converted some logs to their appropriate type (i.e. error to info or vice versa) Reworked schedule print Standardized glog levels `grep -r V\([0-9]\) *` L0: None L1: Highest Level current status info and Errors with Terminations L2: Successful terminations L3: More detailed schedule status info L4: Debugging verbose schedule and config info L5: Auto-resolved inconsequential issues

Allows for fmt.Println(schedule)

Since we should glog the single line info and make the table version isolated and readable

Had to run `glide cache-clear` and then `glide update --strip-vendor`

CreateClient involves creating a clientset that accesses the k8 api It does not belong in chaos but rather kubernetes

New sublocation for all kinds

run with make gofmt

standardized variables to loc added timezone for debugging printing specified format printout for schedule addresses asobti#21 and asobti#23

Allow victims to do the heavylifting for common methods

Aergonus · 2018-01-03T00:05:26Z

Easy extension for #10

Trivial to add new types

See asobti#29

Aergonus added 15 commits November 16, 2017 11:30

Merge remote-tracking branch 'refs/remotes/asobti/master'

93a6b1e

Schedule String()

a1b789e

Allows for fmt.Println(schedule)

Switch Print() and String()

1698d12

Since we should glog the single line info and make the table version isolated and readable

Client-go update

c7976f9

Glide and vendor update

970ed42

Had to run `glide cache-clear` and then `glide update --strip-vendor`

Refactor CreateClient

655399e

CreateClient involves creating a clientset that accesses the k8 api It does not belong in chaos but rather kubernetes

Templatize for victims

ac12218

Move deployments to victims factory

84ca6da

New sublocation for all kinds

Convert deployment to victim template

166a83d

go fmt codebase

b768751

run with make gofmt

Print time fixes

37b4e42

standardized variables to loc added timezone for debugging printing specified format printout for schedule addresses asobti#21 and asobti#23

Refactor deployments --> victims

e43f906

Allow victims to do the heavylifting for common methods

Copy deployments --> statefulsets

dc6ab73

Add statefulsets

9ffa53c

Aergonus mentioned this pull request Jan 3, 2018

Add Whitelist #30

Merged

Aergonus added a commit to Aergonus/kube-monkey that referenced this pull request Jan 3, 2018

Added statefulsets

c70db49

See asobti#29

This was referenced Jan 3, 2018

Statefulsets Aergonus/kube-monkey#9

Merged

Simplified Merge-All #33

Merged

asobti merged commit 9ffa53c into asobti:master Jan 24, 2018

Aergonus deleted the statefulsets branch January 24, 2018 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Statefulsets #29

Add Statefulsets #29

Aergonus commented Jan 3, 2018

Aergonus commented Jan 3, 2018

Add Statefulsets #29

Add Statefulsets #29

Conversation

Aergonus commented Jan 3, 2018

Test SS yaml

Sample output:

Aergonus commented Jan 3, 2018