Metricbeat secrets (#27)

* pass secrets to helm Signed-off-by: yyyogev <[email protected]> * secrets template Signed-off-by: yyyogev <[email protected]> * auto choose https for eks Signed-off-by: yyyogev <[email protected]> * small fixes Signed-off-by: yyyogev <[email protected]> * readme - pass secrets to helm, changelog Signed-off-by: yyyogev <[email protected]> * add secrets to values table Signed-off-by: yyyogev <[email protected]> * minor fix Signed-off-by: yyyogev <[email protected]> * update Chart version Co-authored-by: mirii1994 <[email protected]>
logzio · Nov 25, 2020 · 7b38c5f · 7b38c5f
1 parent 565d9e9
commit 7b38c5f
Show file tree

Hide file tree

Showing 5 changed files with 86 additions and 48 deletions.
diff --git a/metricbeat/Chart.yaml b/metricbeat/Chart.yaml
@@ -1,5 +1,5 @@
 apiVersion: v1
 name: logzio-k8s-metrics
 description: A Helm chart for shipping k8s metrics to logzio.io
-version: 0.0.4
+version: 0.0.5
 appVersion: 7.9.1
diff --git a/metricbeat/README.md b/metricbeat/README.md
@@ -17,7 +17,7 @@ You have two options for deployment:
 * [Automated configuration <span class="sm ital">(recommended)</span>](#default-config)
 * [Manual configuration](#manual-config)
 
-**Note:** Helm 2 will reach [EOL on November 2020](https://helm.sh/blog/2019-10-22-helm-2150-released/#:~:text=6%20months%20after%20Helm%203's,Helm%202%20will%20formally%20end). This document follows the command syntax recommended for Helm 3, but the Chart will work with both Helm 2 and Helm 3. 
+**Note:** Helm 2 will reach [EOL on November 2020](https://helm.sh/blog/2019-10-22-helm-2150-released/#:~:text=6%20months%20after%20Helm%203's,Helm%202%20will%20formally%20end). This document follows the command syntax recommended for Helm 3, but the Chart will work with both Helm 2 and Helm 3.
 
 <div id="default-config">
 
@@ -47,37 +47,15 @@ Give your metrics some time to get from your system to ours, and then open [Logz
 
 ### Manual deployment:
 
-#### 1. Store your Logz.io credentials
-Save your Logz.io shipping credentials as a Kubernetes secret.
 
-Replace `<<SHIPPING-TOKEN>>` with the [token](https://app.logz.io/#/dashboard/settings/general) of the account you want to ship to.
 
-Replace `<<LISTENER-HOST>>` with your region’s listener host (for example, `listener.logz.io`). For more information on finding your account’s region, see [Account region](https://docs.logz.io/user-guide/accounts/account-region.html).
-
-```shell
-kubectl --namespace=kube-system create secret generic logzio-metrics-secret \
-  --from-literal=logzio-metrics-shipping-token=<<SHIPPING-TOKEN>> \
-  --from-literal=logzio-metrics-listener-host=<<LISTENER-HOST>>
-```
-
-#### 2. Store your cluster details
-
-Replace `<<KUBE-STATE-METRICS-NAMESPACE>>`, `<<KUBE-STATE-METRICS-PORT>>`, and `<<CLUSTER-NAME>>` in this command to save your cluster details as a Kubernetes secret.
-
-```shell
-kubectl --namespace=kube-system create secret generic cluster-details \
---from-literal=kube-state-metrics-namespace=<<KUBE-STATE-METRICS-NAMESPACE>> \
---from-literal=kube-state-metrics-port=<<KUBE-STATE-METRICS-PORT>> \
---from-literal=cluster-name=<<CLUSTER-NAME>>
-```
-
-#### 3. Add logzio-k8s-metrics repo to your helm repo list
+#### 1. Add logzio-k8s-metrics repo to your helm repo list
 
 ```shell
 helm repo add logzio-helm https://logzio.github.io/logzio-helm/metricbeat
 ```
 
-#### 4. Deploy
+#### 2. Deploy
 
 You have three options for deployment:
 * [Standard configuration](#standard-config)
@@ -88,8 +66,21 @@ You have three options for deployment:
 <div id="standard-config">
 
 #### Deploy with standard configuration:  
+
+Replace `<<METRICS-TOKEN>>` with the [token](https://app.logz.io/#/dashboard/settings/general) of the account you want to ship to.
+
+Replace `<<LISTENER-HOST>>` with your region’s listener host (for example, `listener.logz.io`). For more information on finding your account’s region, see [Account region](https://docs.logz.io/user-guide/accounts/account-region.html).
+
+Replace `<<KUBE-STATE-METRICS-NAMESPACE>>`, `<<KUBE-STATE-METRICS-PORT>>`, and `<<CLUSTER-NAME>>` in this command to save your cluster details as a Kubernetes secret.
+
 ```shell
-helm install --namespace=kube-system logzio-k8s-metrics logzio-helm/logzio-k8s-metrics
+helm install --namespace=kube-system \
+--set=secrets.MetricsToken=<<METRICS-TOKEN>> \
+--set=secrets.ListenerHost=<<LISTENER-HOST>> \
+--set=secrets.ClusterName=<<CLUSTER-NAME>> \
+--set=secrets.KubeStatNamespace=<<KUBE-STATE-METRICS-NAMESPACE>> \
+--set=secrets.KubeStatPort=<<KUBE-STATE-METRICS-PORT>> \
+logzio-k8s-metrics logzio-helm/logzio-k8s-metrics
 ```
 </div>
 
@@ -101,6 +92,11 @@ This Daemonset's default autodiscover configuration is [hints based](https://www
 ```shell
 helm install --namespace=kube-system \
 --set configType='autodiscover' \
+--set=secrets.MetricsToken=<<METRICS-TOKEN>> \
+--set=secrets.ListenerHost=<<LISTENER-HOST>> \
+--set=secrets.ClusterName=<<CLUSTER-NAME>> \
+--set=secrets.KubeStatNamespace=<<KUBE-STATE-METRICS-NAMESPACE>> \
+--set=secrets.KubeStatPort=<<KUBE-STATE-METRICS-PORT>> \
 logzio-k8s-metrics logzio-helm/logzio-k8s-metrics
 ```
 *For more information about Autodiscover:* [Kubernetes configuration](https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html#_kubernetes)
@@ -113,6 +109,11 @@ logzio-k8s-metrics logzio-helm/logzio-k8s-metrics
 #### Deploy with custom configuration:  
 ```shell
 helm install --namespace=kube-system \
+--set=secrets.MetricsToken=<<METRICS-TOKEN>> \
+--set=secrets.ListenerHost=<<LISTENER-HOST>> \
+--set=secrets.ClusterName=<<CLUSTER-NAME>> \
+--set=secrets.KubeStatNamespace=<<KUBE-STATE-METRICS-NAMESPACE>> \
+--set=secrets.KubeStatPort=<<KUBE-STATE-METRICS-PORT>> \
 --set configType='auto-custom' \
 --set-file metricbeatConfig.autoCustomConfig=/path/to/your/config.yaml \
 logzio-k8s-metrics logzio-helm/logzio-k8s-metrics
@@ -128,7 +129,7 @@ metricbeat.yml: |-
   metricbeat.autodiscover:
     # your autodiscover config
     # ...
-              
+
   processors:
     - add_cloud_metadata: ~
   fields:
@@ -168,6 +169,7 @@ Give your metrics some time to get from your system to ours, and then open [Logz
 | `apiVersions.ServiceAccount` | API version of `serviceaccount.yaml`. | `v1` |
 | `apiVersions.ClusterRole` | API version of `clusterrole.yaml`. | `rbac.authorization.k8s.io/v1beta1` |
 | `apiVersions.ClusterRoleBinding` | API version of `clusterrolebinding.yaml`. | `rbac.authorization.k8s.io/v1beta1` |
+| `apiVersions.Secrets` | API version of `secrets.yaml`. | `v1` |
 | `shippingProtocol` | Shipping protocol. | `http` |
 | `shippingPort` | Shipping port. | `10255` |
 | `serviceAccount.create` | Specifies whether a service account should be created. | `true` |
@@ -201,6 +203,11 @@ Give your metrics some time to get from your system to ours, and then open [Logz
 | `deployment.resources` | Allows you to set the resources for Metricbeat Deployment. | See [values.yaml](https://github.com/logzio/logzio-helm/blob/master/metricbeat/values.yaml). |
 | `deployment.secretMounts` | Allows you to easily mount a secret as a file inside the Deployment Useful for mounting certificates and other secrets. | See [values.yaml](https://github.com/logzio/logzio-helm/blob/master/metricbeat/values.yaml). |
 | `namespace` | Chart's namespace | `kube-system` |
+| `secrets.MetricsToken`| Secret with your [logz.io Metrics token](https://docs.logz.io/user-guide/accounts/finding-your-metrics-account-token/). | `""` |
+| `secrets.ListenerHost`| Secret with your [logz.io listener host](https://docs.logz.io/user-guide/accounts/account-region.html#available-regions). | `""` |
+| `secrets.ClusterName`| Secret with your cluster name. | `""` |
+| `secrets.KubeStatNamespace`| Secret with your Kube-Stat-Metrics namespace. | `""` |
+| `secrets.KubeStatPort`| Secret with your Kube-Stat-Metrics port. | `""` |
 
 If you wish to change the default values, specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
 
@@ -226,6 +233,8 @@ helm uninstall --namespace=kube-system logzio-k8s-metrics
 
 
 ## Change log
+  - **0.0.5**:
+    - Mangage Logz.io metrics related secrets in helm
   - **0.0.4**:
     - Support lean configuration for modules in Deployment and Daemonset to match build-in dashboards in Logz.io.
     - Support custom configuration for modules in Deployment and Daemonset.

diff --git a/metricbeat/templates/secrets.yaml b/metricbeat/templates/secrets.yaml
@@ -0,0 +1,20 @@
+apiVersion: {{ .Values.apiVersions.Secrets }}
+kind: Secret
+metadata:
+  name: logzio-metrics-secret
+  namespace: {{ .Values.namespace }}
+type: Opaque
+stringData:
+  logzio-metrics-shipping-token: {{ .Values.secrets.MetricsToken }}
+  logzio-metrics-listener-host: {{ .Values.secrets.ListenerHost }}
+---
+apiVersion: {{ .Values.apiVersions.Secrets }}
+kind: Secret
+metadata:
+  name: cluster-details
+  namespace: {{ .Values.namespace }}
+type: Opaque
+stringData:
+  kube-state-metrics-namespace: {{ .Values.secrets.KubeStatNamespace }}
+  kube-state-metrics-port: {{ .Values.secrets.KubeStatPort | quote }}
+  cluster-name: {{ .Values.secrets.ClusterName }}
diff --git a/metricbeat/values.yaml b/metricbeat/values.yaml
@@ -10,6 +10,7 @@ apiVersions:
   ServiceAccount: v1
   ClusterRole: rbac.authorization.k8s.io/v1beta1
   ClusterRoleBinding: rbac.authorization.k8s.io/v1beta1
+  Secrets: v1
 
 namespace: kube-system
 
@@ -257,7 +258,7 @@ daemonset:
             - fsstat
           processors:
             - include_fields:
-                fields: ["token","tags","type","@timestamp","event.module","metricset.name","host.name","system.cpu.cores", "system.cpu.iowait.pct", "system.cpu.nice.pct", "system.cpu.softirq.pct", "system.cpu.steal.pct", "system.cpu.system.pct", "system.cpu.user.pct", "system.diskio.io.time", "system.diskio.name", "system.diskio.read.bytes", "system.diskio.write.bytes", "system.filesystem.device", "system.filesystem.used.pct", "system.fsstat.total", "system.load.norm.", "system.memory.free", "system.memory.total", "system.memory.used.bytes", "system.memory.used.pct", "system.network.in.bytes", "system.network.name", "system.network.out.bytes", "system.process.cpu.total.pct", "system.process.memory.rss.bytes", "cluster", "system.load.norm.1", "system.load.1", "system.load.5", "system.load.15", "system.filesystem.used.pct", "system.filesystem.device_name", "system.fsstat.total_size.used", "system.fsstat.total_size.total", "system.cpu.cores", "system.process.cpu.total.pct", "system.process.memory.rss.bytes" , "process.name"]  
+                fields: ["token","tags","type","@timestamp","event.module","metricset.name","host.name","system.cpu.cores", "system.cpu.iowait.pct", "system.cpu.nice.pct", "system.cpu.softirq.pct", "system.cpu.steal.pct", "system.cpu.system.pct", "system.cpu.user.pct", "system.diskio.io.time", "system.diskio.name", "system.diskio.read.bytes", "system.diskio.write.bytes", "system.filesystem.device", "system.filesystem.used.pct", "system.fsstat.total", "system.load.norm.", "system.memory.free", "system.memory.total", "system.memory.used.bytes", "system.memory.used.pct", "system.network.in.bytes", "system.network.name", "system.network.out.bytes", "system.process.cpu.total.pct", "system.process.memory.rss.bytes", "cluster", "system.load.norm.1", "system.load.1", "system.load.5", "system.load.15", "system.filesystem.used.pct", "system.filesystem.device_name", "system.fsstat.total_size.used", "system.fsstat.total_size.total", "system.cpu.cores", "system.process.cpu.total.pct", "system.process.memory.rss.bytes" , "process.name"]
       kubernetes.yml: |-
         - module: kubernetes
           period: 10s
@@ -337,4 +338,10 @@ deployment:
   - name: cert
     mountPath: "/etc/pki/tls/certs/SectigoRSADomainValidationSecureServerCA.crt"
     readOnly: true
-    subPath: SectigoRSADomainValidationSecureServerCA.crt
+    subPath: SectigoRSADomainValidationSecureServerCA.crt
+secrets:
+  MetricsToken: ""
+  ListenerHost: ""
+  ClusterName: ""
+  KubeStatNamespace: ""
+  KubeStatPort: ""
diff --git a/quickstart-metrics.sh b/quickstart-metrics.sh
@@ -67,31 +67,28 @@ else
 fi
 listener_host="listener${logzio_region}.logz.io"
 
-read -ep "Kubelet shipping protocol [http]: " shipping_protocol
-shipping_protocol=${shipping_protocol:-"http"}
-shipping_port="10255"
-if [[ $shipping_protocol == "https" ]]; then
-  shipping_port="10250"
-fi
-
-read -ep "Target namespace to deploy [kube-system]: " namespace
-namespace=${namespace:-"kube-system"}
-
-kubectl --namespace=${namespace} create secret generic logzio-metrics-secret \
-  --from-literal=logzio-metrics-shipping-token=$metrics_token \
-  --from-literal=logzio-metrics-listener-host=$listener_host
-
 cluster_name=$(kubectl config current-context)
 if [[ $cluster_name == *"cluster/"* ]]; then
   cluster_name=${cluster_name#*"cluster/"}
 fi
 read -ep "Cluster name [${cluster_name}]: " real_cluster_name
 real_cluster_name=${real_cluster_name:-"${cluster_name}"}
 
-kubectl --namespace=${namespace} create secret generic cluster-details \
-  --from-literal=kube-state-metrics-namespace=$kube_stat_ns \
-  --from-literal=kube-state-metrics-port=$kube_stat_port \
-  --from-literal=cluster-name=$cluster_name
+has_eks=$(aws eks describe-cluster --name ${real_cluster_name} | grep ":eks:")
+if [[ $has_eks ]]; then
+  shipping_protocol="https"
+else
+  read -ep "Kubelet shipping protocol [http]: " shipping_protocol
+  shipping_protocol=${shipping_protocol:-"http"}
+fi
+
+shipping_port="10255"
+if [[ $shipping_protocol == "https" ]]; then
+  shipping_port="10250"
+fi
+
+read -ep "Target namespace to deploy [kube-system]: " namespace
+namespace=${namespace:-"kube-system"}
 
 read -ep "Deploy with standard or autodiscover configuration? [standard]: " deployment_config
 deployment_config=${deployment_config:-"standard"}
@@ -114,4 +111,9 @@ helm install ${debug} \
 --set=apiVersions.ClusterRole=${clusterrole_api} \
 --set=apiVersions.ClusterRoleBinding=${clusterrolebinding_api} \
 --set=configType=${deployment_config} \
+--set=secrets.MetricsToken=${metrics_token} \
+--set=secrets.ListenerHost=${listener_host} \
+--set=secrets.ClusterName=${real_cluster_name} \
+--set=secrets.KubeStatNamespace=${kube_stat_ns} \
+--set=secrets.KubeStatPort=${kube_stat_port} \
 --repo https://logzio.github.io/logzio-helm/metricbeat logzio-k8s-metrics logzio-k8s-metrics