Metricbeat and Filebeat dont send correct hostname (host.name) #462

talesporto · 2020-01-27T17:31:20Z

Chart version:
7.5.2

Kubernetes version:
v1.14.6

Kubernetes provider:
On-premise installed using RKE

Helm Version:
v2.14.3

helm get release output

Output of helm get elasticsearch

REVISION: 2
RELEASED: Fri Jan 24 17:07:24 2020
CHART: elasticsearch-7.5.2
USER-SUPPLIED VALUES:
esConfig:
  elasticsearch.yml: "path.repo: /tmp\nhttp.cors.enabled: true\nhttp.cors.allow-origin:
    \"*\"\nhttp.cors.allow-headers: \"X-Requested-With, Content-Type, Content-Length,
    Authorization\"\nxpack.monitoring.enabled: true\nxpack.monitoring.collection.enabled:
    true\nxpack.security.enabled: true\nxpack.security.transport.ssl.enabled: true\nxpack.security.transport.ssl.verification_mode:
    certificate\nxpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key\nxpack.security.transport.ssl.certificate:
    /usr/share/elasticsearch/config/certs/ca.crt\nxpack.security.http.ssl.enabled:
    true\nxpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key\nxpack.security.http.ssl.certificate:
    /usr/share/elasticsearch/config/certs/ca.crt\nxpack.security.authc:\n  anonymous:\n
    \   username: anonymous_user \n    roles: watcher_user\nxpack.security.authc.realms:\n
    \ native.realm1:\n    order: 0\n  file.realm2:\n    order: 1\n    \n"
extraEnvs:
- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
image: registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/elasticsearch
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2-1
nodeAffinity:
  env.infra: "true"
protocol: https
replicas: 3
secretMounts:
- name: elastic-certificates
  path: /usr/share/elasticsearch/config/certs
  secretName: elastic-certificates
volumeClaimTemplate:
  resources:
    requests:
      storage: 50G
COMPUTED VALUES:
antiAffinity: hard
antiAffinityTopologyKey: kubernetes.io/hostname
clusterHealthCheckParams: wait_for_status=green&timeout=1s
clusterName: elasticsearch
esConfig:
  elasticsearch.yml: "path.repo: /tmp\nhttp.cors.enabled: true\nhttp.cors.allow-origin:
    \"*\"\nhttp.cors.allow-headers: \"X-Requested-With, Content-Type, Content-Length,
    Authorization\"\nxpack.monitoring.enabled: true\nxpack.monitoring.collection.enabled:
    true\nxpack.security.enabled: true\nxpack.security.transport.ssl.enabled: true\nxpack.security.transport.ssl.verification_mode:
    certificate\nxpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key\nxpack.security.transport.ssl.certificate:
    /usr/share/elasticsearch/config/certs/ca.crt\nxpack.security.http.ssl.enabled:
    true\nxpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key\nxpack.security.http.ssl.certificate:
    /usr/share/elasticsearch/config/certs/ca.crt\nxpack.security.authc:\n  anonymous:\n
    \   username: anonymous_user \n    roles: watcher_user\nxpack.security.authc.realms:\n
    \ native.realm1:\n    order: 0\n  file.realm2:\n    order: 1\n    \n"
esJavaOpts: -Xmx1g -Xms1g
esMajorVersion: ""
extraEnvs:
- name: ELASTIC_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
- name: ELASTIC_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
extraInitContainers: ""
extraVolumeMounts: ""
extraVolumes: ""
fsGroup: ""
fullnameOverride: ""
httpPort: 9200
image: registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/elasticsearch
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2-1
ingress:
  annotations: {}
  enabled: false
  hosts:
  - chart-example.local
  path: /
  tls: []
initResources: {}
keystore: []
labels: {}
lifecycle: {}
masterService: ""
masterTerminationFix: false
maxUnavailable: 1
minimumMasterNodes: 2
nameOverride: ""
networkHost: 0.0.0.0
nodeAffinity:
  env.infra: "true"
nodeGroup: master
nodeSelector: {}
persistence:
  annotations: {}
  enabled: true
podAnnotations: {}
podManagementPolicy: Parallel
podSecurityContext:
  fsGroup: 1000
  runAsUser: 1000
podSecurityPolicy:
  create: false
  name: ""
  spec:
    fsGroup:
      rule: RunAsAny
    privileged: true
    runAsUser:
      rule: RunAsAny
    seLinux:
      rule: RunAsAny
    supplementalGroups:
      rule: RunAsAny
    volumes:
    - secret
    - configMap
    - persistentVolumeClaim
priorityClassName: ""
protocol: https
rbac:
  create: false
  serviceAccountName: ""
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 3
resources:
  limits:
    cpu: 1000m
    memory: 2Gi
  requests:
    cpu: 100m
    memory: 2Gi
roles:
  data: "true"
  ingest: "true"
  master: "true"
schedulerName: ""
secretMounts:
- name: elastic-certificates
  path: /usr/share/elasticsearch/config/certs
  secretName: elastic-certificates
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
service:
  annotations: {}
  httpPortName: http
  labels: {}
  labelsHeadless: {}
  nodePort: ""
  transportPortName: transport
  type: ClusterIP
sidecarResources: {}
sysctlInitContainer:
  enabled: true
sysctlVmMaxMapCount: 262144
terminationGracePeriod: 120
tolerations: []
transportPort: 9300
updateStrategy: RollingUpdate
volumeClaimTemplate:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 50G
HOOKS:
---
# elasticsearch-lzure-test
apiVersion: v1
kind: Pod
metadata:
  name: "elasticsearch-lzure-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
  - name: "elasticsearch-gacef-test"
    image: "registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/elasticsearch:7.5.2-1"
    command:
      - "sh"
      - "-c"
      - |
        #!/usr/bin/env bash -e
        curl -XGET --fail 'elasticsearch-master:9200/_cluster/health?wait_for_status=green&timeout=1s'
  restartPolicy: Never
MANIFEST:
---
# Source: elasticsearch/templates/poddisruptionbudget.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: "elasticsearch-master-pdb"
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: "elasticsearch-master"
---
# Source: elasticsearch/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-master-config
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
data:
  elasticsearch.yml: |
    path.repo: /tmp
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    http.cors.allow-headers: "X-Requested-With, Content-Type, Content-Length, Authorization"
    xpack.monitoring.enabled: true
    xpack.monitoring.collection.enabled: true
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/certs/ca.crt
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.key: /usr/share/elasticsearch/config/certs/tls.key
    xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/certs/ca.crt
    xpack.security.authc:
      anonymous:
        username: anonymous_user 
        roles: watcher_user
    xpack.security.authc.realms:
      native.realm1:
        order: 0
      file.realm2:
        order: 1
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master-headless
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
spec:
  clusterIP: None # This is needed for statefulset hostnames like elasticsearch-0 to resolve
  # Create endpoints also if the related pod isn't ready
  publishNotReadyAddresses: true
  selector:
    app: "elasticsearch-master"
  ports:
  - name: http
    port: 9200
  - name: transport
    port: 9300
---
# Source: elasticsearch/templates/service.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    {}
spec:
  type: ClusterIP
  selector:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  ports:
  - name: http
    protocol: TCP
    port: 9200
  - name: transport
    protocol: TCP
    port: 9300
---
# Source: elasticsearch/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-master
  labels:
    heritage: "Tiller"
    release: "elasticsearch"
    chart: "elasticsearch"
    app: "elasticsearch-master"
  annotations:
    esMajorVersion: "7"
spec:
  serviceName: elasticsearch-master-headless
  selector:
    matchLabels:
      app: "elasticsearch-master"
  replicas: 3
  podManagementPolicy: Parallel
  updateStrategy:
    type: RollingUpdate
  volumeClaimTemplates:
  - metadata:
      name: elasticsearch-master
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 50G
  template:
    metadata:
      name: "elasticsearch-master"
      labels:
        heritage: "Tiller"
        release: "elasticsearch"
        chart: "elasticsearch"
        app: "elasticsearch-master"
      annotations:        
        configchecksum: d1d8755529f95d923821e30573ac3ff8d122078d617723895cbeb8f95f79785
    spec:
      securityContext:
        fsGroup: 1000
        runAsUser: 1000        
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - "elasticsearch-master"
            topologyKey: kubernetes.io/hostname
        nodeAffinity:
          env.infra: "true"          
      terminationGracePeriodSeconds: 120
      volumes:
        - name: elastic-certificates
          secret:
            secretName: elastic-certificates
        - name: esconfig
          configMap:
            name: elasticsearch-master-config
      imagePullSecrets:
        - name: gitlab-docker-registry-credential        
      initContainers:
      - name: configure-sysctl
        securityContext:
          runAsUser: 0
          privileged: true
        image: "registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/elasticsearch:7.5.2-1"
        imagePullPolicy: "IfNotPresent"
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        resources:
          {}
      containers:
      - name: "elasticsearch"
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000          
        image: "registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/elasticsearch:7.5.2-1"
        imagePullPolicy: "IfNotPresent"
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5          
          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
                # Once it has started only check that the node itself is responding
                START_FILE=/tmp/.es_start_file
                http () {
                    local path="${1}"
                    if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                      BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                    else
                      BASIC_AUTH=''
                    fi
                    curl -XGET -s -k --fail ${BASIC_AUTH} https://127.0.0.1:9200${path}
                }
                if [ -f "${START_FILE}" ]; then
                    echo 'Elasticsearch is already running, lets check the node is healthy and there are master nodes available'
                    http "/_cluster/health?timeout=0s"
                else
                    echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                    if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                        touch ${START_FILE}
                        exit 0
                    else
                        echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                        exit 1
                    fi
                fi
        ports:
        - name: http
          containerPort: 9200
        - name: transport
          containerPort: 9300
        resources:
          limits:
            cpu: 1000m
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 2Gi          
        env:
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: cluster.initial_master_nodes
            value: "elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,"
          - name: discovery.seed_hosts
            value: "elasticsearch-master-headless"
          - name: cluster.name
            value: "elasticsearch"
          - name: network.host
            value: "0.0.0.0"
          - name: ES_JAVA_OPTS
            value: "-Xmx1g -Xms1g"
          - name: node.data
            value: "true"
          - name: node.ingest
            value: "true"
          - name: node.master
            value: "true"
          - name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: elastic-credentials
          - name: ELASTIC_USERNAME
            valueFrom:
              secretKeyRef:
                key: username
                name: elastic-credentials          
        volumeMounts:
          - name: "elasticsearch-master"
            mountPath: /usr/share/elasticsearch/data
          - name: elastic-certificates
            mountPath: /usr/share/elasticsearch/config/certs
          - name: esconfig
            mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
            subPath: elasticsearch.yml

Output of helm get kibana

REVISION: 1
RELEASED: Fri Jan 24 17:19:26 2020
CHART: kibana-7.5.2
USER-SUPPLIED VALUES:
elasticsearchHosts: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2
ingress:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  enabled: true
  hosts:
  - kibana.onstreet.exploraparking.com.br
  path: /
  tls:
  - hosts:
    - kibana.onstreet.exploraparking.com.br
    secretName: tls-kibana-ingress
kibanaConfig:
  kibana.yml: |
    xpack.security.encryptionKey: DV2ccA7hC@Lbd-!BKcBuGcg8yyVp7M*FunfmQkebGKKnZXqkUmJogzoRrWrw
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/ca.crt
      verificationMode: certificate
nodeAffinity:
  env.infra: "true"
secretMounts:
- name: elastic-certificates
  path: /usr/share/kibana/config/certs
  secretName: elastic-certificates

COMPUTED VALUES:
affinity: {}
elasticsearchHosts: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
elasticsearchURL: ""
extraEnvs:
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
fullnameOverride: ""
healthCheckPath: /app/kibana
httpPort: 5601
image: docker.elastic.co/kibana/kibana
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2
ingress:
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  enabled: true
  hosts:
  - kibana.onstreet.exploraparking.com.br
  path: /
  tls:
  - hosts:
    - kibana.onstreet.exploraparking.com.br
    secretName: tls-kibana-ingress
kibanaConfig:
  kibana.yml: |
    xpack.security.encryptionKey: DV2ccA7hC@Lbd-!BKcBuGcg8yyVp7M*FunfmQkebGKKnZXqkUmJogzoRrWrw
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/ca.crt
      verificationMode: certificate
labels: {}
lifecycle: {}
nameOverride: ""
nodeAffinity:
  env.infra: "true"
nodeSelector: {}
podAnnotations: {}
podSecurityContext:
  fsGroup: 1000
priorityClassName: ""
protocol: http
readinessProbe:
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  successThreshold: 3
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 1000m
    memory: 1Gi
  requests:
    cpu: 100m
    memory: 500Mi
secretMounts:
- name: elastic-certificates
  path: /usr/share/kibana/config/certs
  secretName: elastic-certificates
securityContext:
  capabilities:
    drop:
    - ALL
  runAsNonRoot: true
  runAsUser: 1000
serverHost: 0.0.0.0
service:
  annotations: {}
  labels: {}
  loadBalancerSourceRanges: []
  nodePort: ""
  port: 5601
  type: ClusterIP
serviceAccount: ""
tolerations: []
updateStrategy:
  type: Recreate

HOOKS:
MANIFEST:

---
# Source: kibana/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-kibana-config
  labels:
    app: kibana
    release: "kibana"
data:
  kibana.yml: |
    xpack.security.encryptionKey: DV2ccA7hC@Lbd-!BKcBuGcg8yyVp7M*FunfmQkebGKKnZXqkUmJogzoRrWrw
    elasticsearch.ssl:
      certificateAuthorities: /usr/share/kibana/config/certs/ca.crt
      verificationMode: certificate
---
# Source: kibana/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: "kibana"
    heritage: Tiller
spec:
  type: ClusterIP
  ports:
    - port: 5601
      protocol: TCP
      name: http
      targetPort: 5601
  selector:
    app: kibana
    release: "kibana"
---
# Source: kibana/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: "kibana"
spec:
  replicas: 1
  strategy:
    type: Recreate
    
  selector:
    matchLabels:
      app: kibana
      release: "kibana"
  template:
    metadata:
      labels:
        app: kibana
        release: "kibana"
      annotations:
        
        configchecksum: 6c0c0da88225f4f030cb5617b1cedc4fe342b2fc8d25bdadff8e8cf259a4597
    spec:
      securityContext:
        fsGroup: 1000
        
      volumes:
        - name: elastic-certificates
          secret:
            secretName: elastic-certificates
        - name: kibanaconfig
          configMap:
            name: kibana-kibana-config
      imagePullSecrets:
        - name: gitlab-docker-registry-credential
        
      containers:
      - name: kibana
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
          
        image: "docker.elastic.co/kibana/kibana:7.5.2"
        imagePullPolicy: "IfNotPresent"
        env:
          - name: ELASTICSEARCH_HOSTS
            value: "https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200"
          - name: SERVER_HOST
            value: "0.0.0.0"
          - name: ELASTICSEARCH_USERNAME
            valueFrom:
              secretKeyRef:
                key: username
                name: elastic-credentials
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: elastic-credentials
          
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
          
          exec:
            command:
              - sh
              - -c
              - |
                #!/usr/bin/env bash -e
                http () {
                    local path="${1}"
                    set -- -XGET -s --fail

                    if [ -n "${ELASTICSEARCH_USERNAME}" ] && [ -n "${ELASTICSEARCH_PASSWORD}" ]; then
                      set -- "$@" -u "${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD}"
                    fi

                    STATUS=$(curl --output /dev/null --write-out "%{http_code}" -k "$@" "http://localhost:5601${path}")
                    if [[ "${STATUS}" -eq 200 ]]; then
                      exit 0
                    fi

                    echo "Error: Got HTTP code ${STATUS} but expected a 200"
                    exit 1
                }

                http "/app/kibana"
        ports:
        - containerPort: 5601
        resources:
          limits:
            cpu: 1000m
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 500Mi
          
        volumeMounts:
          - name: elastic-certificates
            mountPath: /usr/share/kibana/config/certs
          - name: kibanaconfig
            mountPath: /usr/share/kibana/config/kibana.yml
            subPath: kibana.yml
---
# Source: kibana/templates/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: kibana-kibana
  labels:
    app: kibana
    release: kibana
    heritage: Tiller
  annotations:
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    
spec:
  tls:
    - hosts:
      - kibana.onstreet.exploraparking.com.br
      secretName: tls-kibana-ingress
    
  rules:
    - host: kibana.onstreet.exploraparking.com.br
      http:
        paths:
          - path: /
            backend:
              serviceName: kibana-kibana
              servicePort: 5601

Output of helm get filebeat

REVISION: 13
RELEASED: Mon Jan 27 13:56:35 2020
CHART: filebeat-7.5.2
USER-SUPPLIED VALUES:
extraEnvs:
- name: ELASTICSEARCH_HOSTS
  value: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
filebeatConfig:
  filebeat.yml: |
    logging.level: info

    setup.kibana.host: 'http://kibana-kibana.mobvaga-monitoring.svc.cluster.local:5601'
    # setup.dashboards.enabled: true
    # setup.dashboards.always_kibana: true
    # setup.dashboards.retry.enabled: true

    filebeat.config:
      modules:
        - module: nginx
          access:
            enabled: true
          error:
            enabled: true

    # filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/log/containers/*.log
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config.enabled: false
          hints.default_config:
            type: docker
            containers.ids:
              - '${data.kubernetes.container.id}'
            processors:
              - add_kubernetes_metadata:
                  in_cluster: true
            # type: container
            # paths:
            #   - /var/log/containers/*${data.kubernetes.container.id}.log

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}']
      ssl.certificate_authorities: ["/usr/share/filebeat/config/certs/ca.crt"]
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
image: registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/filebeat
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2-1
nodeAffinity:
  env.infra: "true"
secretMounts:
- name: elastic-certificates
  path: /usr/share/filebeat/config/certs
  secretName: elastic-certificates

COMPUTED VALUES:
affinity: {}
envFrom: []
extraEnvs:
- name: ELASTICSEARCH_HOSTS
  value: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
extraVolumeMounts: []
extraVolumes: []
filebeatConfig:
  filebeat.yml: |
    logging.level: info

    setup.kibana.host: 'http://kibana-kibana.mobvaga-monitoring.svc.cluster.local:5601'
    # setup.dashboards.enabled: true
    # setup.dashboards.always_kibana: true
    # setup.dashboards.retry.enabled: true

    filebeat.config:
      modules:
        - module: nginx
          access:
            enabled: true
          error:
            enabled: true

    # filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/log/containers/*.log
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"

    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config.enabled: false
          hints.default_config:
            type: docker
            containers.ids:
              - '${data.kubernetes.container.id}'
            processors:
              - add_kubernetes_metadata:
                  in_cluster: true
            # type: container
            # paths:
            #   - /var/log/containers/*${data.kubernetes.container.id}.log

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}']
      ssl.certificate_authorities: ["/usr/share/filebeat/config/certs/ca.crt"]
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
fullnameOverride: ""
hostNetworking: false
hostPathRoot: /var/lib
image: registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/filebeat
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2-1
labels: {}
livenessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      curl --fail 127.0.0.1:5066
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
managedServiceAccount: true
nameOverride: ""
nodeAffinity:
  env.infra: "true"
nodeSelector: {}
podAnnotations: {}
podSecurityContext:
  privileged: false
  runAsUser: 0
priorityClassName: ""
readinessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      filebeat test output
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
resources:
  limits:
    cpu: 1000m
    memory: 200Mi
  requests:
    cpu: 100m
    memory: 100Mi
secretMounts:
- name: elastic-certificates
  path: /usr/share/filebeat/config/certs
  secretName: elastic-certificates
serviceAccount: ""
terminationGracePeriod: 30
tolerations: []
updateStrategy: RollingUpdate

HOOKS:
MANIFEST:

---
# Source: filebeat/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-filebeat-config
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.5.2"
    heritage: "Tiller"
    release: "filebeat"
data:
  filebeat.yml: |
    logging.level: info
    
    setup.kibana.host: 'http://kibana-kibana.mobvaga-monitoring.svc.cluster.local:5601'
    # setup.dashboards.enabled: true
    # setup.dashboards.always_kibana: true
    # setup.dashboards.retry.enabled: true
    
    filebeat.config:
      modules:
        - module: nginx
          access:
            enabled: true
          error:
            enabled: true
    
    # filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/log/containers/*.log
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"
    
    # To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config.enabled: false
          hints.default_config:
            type: docker
            containers.ids:
              - '${data.kubernetes.container.id}'
            processors:
              - add_kubernetes_metadata:
                  in_cluster: true
            # type: container
            # paths:
            #   - /var/log/containers/*${data.kubernetes.container.id}.log
    
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}']
      ssl.certificate_authorities: ["/usr/share/filebeat/config/certs/ca.crt"]
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---
# Source: filebeat/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat-filebeat
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.5.2"
    heritage: "Tiller"
    release: "filebeat"
---
# Source: filebeat/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat-filebeat-cluster-role
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.5.2"
    heritage: "Tiller"
    release: "filebeat"
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - list
  - watch
---
# Source: filebeat/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat-filebeat-cluster-role-binding
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.5.2"
    heritage: "Tiller"
    release: "filebeat"
roleRef:
  kind: ClusterRole
  name: filebeat-filebeat-cluster-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: filebeat-filebeat
  namespace: mobvaga-monitoring
---
# Source: filebeat/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat-filebeat
  labels:
    app: "filebeat-filebeat"
    chart: "filebeat-7.5.2"
    heritage: "Tiller"
    release: "filebeat"
spec:
  selector:
    matchLabels:
      app: "filebeat-filebeat"
      release: "filebeat"
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        
        configChecksum: 05a707ad80f0479af5e4e779e7246844f21c9160fa4312308b26405283f42cb
      name: "filebeat-filebeat"
      labels:
        app: "filebeat-filebeat"
        chart: "filebeat-7.5.2"
        heritage: "Tiller"
        release: "filebeat"
    spec:
      serviceAccountName: filebeat-filebeat
      terminationGracePeriodSeconds: 30
      volumes:
      - name: elastic-certificates
        secret:
          secretName: elastic-certificates
      - name: filebeat-config
        configMap:
          defaultMode: 0600
          name: filebeat-filebeat-config
      - name: data
        hostPath:
          path: /var/lib/filebeat-filebeat-mobvaga-monitoring-data
          type: DirectoryOrCreate
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varrundockersock
        hostPath:
          path: /var/run/docker.sock
      imagePullSecrets:
        - name: gitlab-docker-registry-credential
        
      containers:
      - name: "filebeat"
        image: "registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/filebeat:7.5.2-1"
        imagePullPolicy: "IfNotPresent"
        args:
        - "-e"
        - "-E"
        - "http.enabled=true"
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              curl --fail 127.0.0.1:5066
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              filebeat test output
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          
        resources:
          limits:
            cpu: 1000m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
          
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: ELASTICSEARCH_HOSTS
          value: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
        - name: ELASTICSEARCH_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: elastic-credentials
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: elastic-credentials
        
        securityContext:
          privileged: false
          runAsUser: 0
          
        volumeMounts:
        - name: elastic-certificates
          mountPath: /usr/share/filebeat/config/certs
        - name: filebeat-config
          mountPath: /usr/share/filebeat/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        # Necessary when using autodiscovery; avoid mounting it otherwise
        # See: https://www.elastic.co/guide/en/beats/filebeat/master/configuration-autodiscover.html
        - name: varrundockersock
          mountPath: /var/run/docker.sock
          readOnly: true

Output of helm get metricbeat

REVISION: 9
RELEASED: Mon Jan 27 13:56:35 2020
CHART: metricbeat-7.5.2
USER-SUPPLIED VALUES:
clusterRoleRules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
extraEnvs:
- name: ELASTICSEARCH_HOSTS
  value: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
image: registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/metricbeat
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2-1
managedServiceAccount: true
metricbeatConfig:
  kube-state-metrics-metricbeat.yml: |
    metricbeat.modules:
    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
      period: 10s
      host: ${NODE_NAME}
      hosts: ["${KUBE_STATE_METRICS_HOSTS}"]

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca.crt"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
  metricbeat.yml: |
    setup.kibana.host: "http://kibana-kibana.mobvaga-monitoring.svc.cluster.local:5601"
    # setup.dashboards.enabled: true
    # setup.dashboards.always_kibana: true
    # setup.dashboards.retry.enabled: true

    metricbeat.modules:
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      host: "${NODE_NAME}"
      hosts: ["https://${NODE_NAME}:10250"]
      ssl.verification_mode: none
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      processors:
      - add_kubernetes_metadata:
          in_cluster: true

    - module: kubernetes
      enabled: true
      metricsets:
        - event
      host: ${NODE_NAME}

    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5
        by_memory: 5

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca.crt"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
nodeAffinity:
  env.infra: "true"
secretMounts:
- name: elastic-certificates
  path: /usr/share/metricbeat/config/certs
  secretName: elastic-certificates

COMPUTED VALUES:
affinity: {}
clusterRoleRules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
envFrom: []
extraEnvs:
- name: ELASTICSEARCH_HOSTS
  value: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
- name: ELASTICSEARCH_USERNAME
  valueFrom:
    secretKeyRef:
      key: username
      name: elastic-credentials
- name: ELASTICSEARCH_PASSWORD
  valueFrom:
    secretKeyRef:
      key: password
      name: elastic-credentials
extraVolumeMounts: []
extraVolumes: []
fullnameOverride: ""
hostPathRoot: /var/lib
image: registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/metricbeat
imagePullPolicy: IfNotPresent
imagePullSecrets:
- name: gitlab-docker-registry-credential
imageTag: 7.5.2-1
kube-state-metrics:
  affinity: {}
  collectors:
    certificatesigningrequests: true
    configmaps: true
    cronjobs: true
    daemonsets: true
    deployments: true
    endpoints: true
    horizontalpodautoscalers: true
    ingresses: true
    jobs: true
    limitranges: true
    namespaces: true
    nodes: true
    persistentvolumeclaims: true
    persistentvolumes: true
    poddisruptionbudgets: true
    pods: true
    replicasets: true
    replicationcontrollers: true
    resourcequotas: true
    secrets: true
    services: true
    statefulsets: true
    storageclasses: true
    verticalpodautoscalers: false
  customLabels: {}
  global: {}
  hostNetwork: false
  image:
    pullPolicy: IfNotPresent
    repository: quay.io/coreos/kube-state-metrics
    tag: v1.8.0
  nodeSelector: {}
  podAnnotations: {}
  podSecurityPolicy:
    annotations: {}
    enabled: false
  prometheus:
    monitor:
      additionalLabels: {}
      enabled: false
      honorLabels: false
      namespace: ""
  prometheusScrape: true
  rbac:
    create: true
  replicas: 1
  securityContext:
    enabled: true
    fsGroup: 65534
    runAsUser: 65534
  service:
    annotations: {}
    loadBalancerIP: ""
    nodePort: 0
    port: 8080
    type: ClusterIP
  serviceAccount:
    create: true
    imagePullSecrets: []
  tolerations: []
labels: {}
livenessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      curl --fail 127.0.0.1:5066
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
managedServiceAccount: true
metricbeatConfig:
  kube-state-metrics-metricbeat.yml: |
    metricbeat.modules:
    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
      period: 10s
      host: ${NODE_NAME}
      hosts: ["${KUBE_STATE_METRICS_HOSTS}"]

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca.crt"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
  metricbeat.yml: |
    setup.kibana.host: "http://kibana-kibana.mobvaga-monitoring.svc.cluster.local:5601"
    # setup.dashboards.enabled: true
    # setup.dashboards.always_kibana: true
    # setup.dashboards.retry.enabled: true

    metricbeat.modules:
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      host: "${NODE_NAME}"
      hosts: ["https://${NODE_NAME}:10250"]
      ssl.verification_mode: none
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      processors:
      - add_kubernetes_metadata:
          in_cluster: true

    - module: kubernetes
      enabled: true
      metricsets:
        - event
      host: ${NODE_NAME}

    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5
        by_memory: 5

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca.crt"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
nameOverride: ""
nodeAffinity:
  env.infra: "true"
nodeSelector: {}
podAnnotations: {}
podSecurityContext:
  privileged: false
  runAsUser: 0
priorityClassName: ""
readinessProbe:
  exec:
    command:
    - sh
    - -c
    - |
      #!/usr/bin/env bash -e
      metricbeat test output
  failureThreshold: 3
  initialDelaySeconds: 10
  periodSeconds: 10
  timeoutSeconds: 5
replicas: 1
resources:
  limits:
    cpu: 1000m
    memory: 200Mi
  requests:
    cpu: 100m
    memory: 100Mi
secretMounts:
- name: elastic-certificates
  path: /usr/share/metricbeat/config/certs
  secretName: elastic-certificates
serviceAccount: ""
terminationGracePeriod: 30
tolerations: []
updateStrategy: RollingUpdate

HOOKS:
MANIFEST:

---
# Source: metricbeat/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-metricbeat-config
  labels:
    app: "metricbeat-metricbeat"
    chart: "metricbeat-7.5.2"
    heritage: "Tiller"
    release: "metricbeat"
data:
  kube-state-metrics-metricbeat.yml: |
    metricbeat.modules:
    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        - state_cronjob
        - state_resourcequota
      period: 10s
      host: ${NODE_NAME}
      hosts: ["${KUBE_STATE_METRICS_HOSTS}"]
    
    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca.crt"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
    
  metricbeat.yml: |
    setup.kibana.host: "http://kibana-kibana.mobvaga-monitoring.svc.cluster.local:5601"
    # setup.dashboards.enabled: true
    # setup.dashboards.always_kibana: true
    # setup.dashboards.retry.enabled: true
    
    metricbeat.modules:
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      host: "${NODE_NAME}"
      hosts: ["https://${NODE_NAME}:10250"]
      ssl.verification_mode: none
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      processors:
      - add_kubernetes_metadata:
          in_cluster: true
    
    - module: kubernetes
      enabled: true
      metricsets:
        - event
      host: ${NODE_NAME}
    
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5
        by_memory: 5
    
    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
    
    output.elasticsearch:
      hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
      ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca.crt"]
      username: "${ELASTICSEARCH_USERNAME}"
      password: "${ELASTICSEARCH_PASSWORD}"
---
# Source: metricbeat/charts/kube-state-metrics/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    helm.sh/chart: kube-state-metrics-2.4.1
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/instance: metricbeat
  name: metricbeat-kube-state-metrics
imagePullSecrets:
  []
---
# Source: metricbeat/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat-metricbeat
  labels:
    app: "metricbeat-metricbeat"
    chart: "metricbeat-7.5.2"
    heritage: "Tiller"
    release: "metricbeat"
---
# Source: metricbeat/charts/kube-state-metrics/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    helm.sh/chart: kube-state-metrics-2.4.1
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/instance: metricbeat
  name: metricbeat-kube-state-metrics
rules:

- apiGroups: ["certificates.k8s.io"]
  resources:
  - certificatesigningrequests
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["list", "watch"]

- apiGroups: ["batch"]
  resources:
  - cronjobs
  verbs: ["list", "watch"]

- apiGroups: ["extensions", "apps"]
  resources:
  - daemonsets
  verbs: ["list", "watch"]

- apiGroups: ["extensions", "apps"]
  resources:
  - deployments
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - endpoints
  verbs: ["list", "watch"]

- apiGroups: ["autoscaling"]
  resources:
  - horizontalpodautoscalers
  verbs: ["list", "watch"]

- apiGroups: ["extensions", "networking.k8s.io"]
  resources:
  - ingresses
  verbs: ["list", "watch"]

- apiGroups: ["batch"]
  resources:
  - jobs
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - limitranges
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - namespaces
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - nodes
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - persistentvolumeclaims
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - persistentvolumes
  verbs: ["list", "watch"]

- apiGroups: ["policy"]
  resources:
    - poddisruptionbudgets
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - pods
  verbs: ["list", "watch"]

- apiGroups: ["extensions", "apps"]
  resources:
  - replicasets
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - replicationcontrollers
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - resourcequotas
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - secrets
  verbs: ["list", "watch"]

- apiGroups: [""]
  resources:
  - services
  verbs: ["list", "watch"]

- apiGroups: ["apps"]
  resources:
  - statefulsets
  verbs: ["list", "watch"]

- apiGroups: ["storage.k8s.io"]
  resources:
    - storageclasses
  verbs: ["list", "watch"]
---
# Source: metricbeat/templates/clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: metricbeat-metricbeat-cluster-role
  labels:
    app: "metricbeat-metricbeat"
    chart: "metricbeat-7.5.2"
    heritage: "Tiller"
    release: "metricbeat"
rules: 
  - apiGroups:
    - ""
    resources:
    - nodes
    - namespaces
    - events
    - pods
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - extensions
    resources:
    - replicasets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - apps
    resources:
    - statefulsets
    - deployments
    - replicasets
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - nodes/stats
    verbs:
    - get
---
# Source: metricbeat/charts/kube-state-metrics/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  labels:
    app.kubernetes.io/name: kube-state-metrics
    helm.sh/chart: kube-state-metrics-2.4.1
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/instance: metricbeat
  name: metricbeat-kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metricbeat-kube-state-metrics
subjects:
- kind: ServiceAccount
  name: metricbeat-kube-state-metrics
  namespace: mobvaga-monitoring
---
# Source: metricbeat/templates/clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metricbeat-metricbeat-cluster-role-binding
  labels:
    app: "metricbeat-metricbeat"
    chart: "metricbeat-7.5.2"
    heritage: "Tiller"
    release: "metricbeat"
roleRef:
  kind: ClusterRole
  name: metricbeat-metricbeat-cluster-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: metricbeat-metricbeat
  namespace: mobvaga-monitoring
---
# Source: metricbeat/charts/kube-state-metrics/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: metricbeat-kube-state-metrics
  labels:
    app.kubernetes.io/name: kube-state-metrics
    helm.sh/chart: "kube-state-metrics-2.4.1"
    app.kubernetes.io/instance: "metricbeat"
    app.kubernetes.io/managed-by: "Tiller"
  annotations:
    prometheus.io/scrape: 'true'
spec:
  type: "ClusterIP"
  ports:
  - name: "http"
    protocol: TCP
    port: 8080
    targetPort: 8080
  selector:
    app.kubernetes.io/name: kube-state-metrics
    app.kubernetes.io/instance: metricbeat
---
# Source: metricbeat/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: metricbeat-metricbeat
  labels:
    app: "metricbeat-metricbeat"
    chart: "metricbeat-7.5.2"
    heritage: "Tiller"
    release: "metricbeat"
spec:
  selector:
    matchLabels:
      app: "metricbeat-metricbeat"
      release: "metricbeat"
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        
        configChecksum: f97bed025a57e28c0a7779e8af6bfc4da29d2dd20a660e480214637511beedc
      name: "metricbeat-metricbeat"
      labels:
        app: "metricbeat-metricbeat"
        chart: "metricbeat-7.5.2"
        heritage: "Tiller"
        release: "metricbeat"
    spec:
      serviceAccountName: metricbeat-metricbeat
      terminationGracePeriodSeconds: 30
      volumes:
      - name: elastic-certificates
        secret:
          secretName: elastic-certificates
      - name: metricbeat-config
        configMap:
          defaultMode: 0600
          name: metricbeat-metricbeat-config
      - name: data
        hostPath:
          path: /var/lib/metricbeat-metricbeat-mobvaga-monitoring-data
          type: DirectoryOrCreate
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: varrundockersock
        hostPath:
          path: /var/run/docker.sock
      - name: proc
        hostPath:
          path: /proc
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
      imagePullSecrets:
        - name: gitlab-docker-registry-credential
        
      containers:
      - name: "metricbeat"
        image: "registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/metricbeat:7.5.2-1"
        imagePullPolicy: "IfNotPresent"
        args:
        - "-e"
        - "-E"
        - "http.enabled=true"
        - "--system.hostfs=/hostfs"
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              curl --fail 127.0.0.1:5066
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              metricbeat test output
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          
        resources:
          limits:
            cpu: 1000m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
          
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: ELASTICSEARCH_HOSTS
          value: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
        - name: ELASTICSEARCH_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: elastic-credentials
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: elastic-credentials
        
        securityContext:
          privileged: false
          runAsUser: 0
          
        volumeMounts:
        - name: elastic-certificates
          mountPath: /usr/share/metricbeat/config/certs
        - name: metricbeat-config
          mountPath: /usr/share/metricbeat/kube-state-metrics-metricbeat.yml
          readOnly: true
          subPath: kube-state-metrics-metricbeat.yml
        - name: metricbeat-config
          mountPath: /usr/share/metricbeat/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: data
          mountPath: /usr/share/metricbeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        # Necessary when using autodiscovery; avoid mounting it otherwise
        # See: https://www.elastic.co/guide/en/beats/metricbeat/master/configuration-autodiscover.html
        - name: varrundockersock
          mountPath: /var/run/docker.sock
          readOnly: true
        - name: proc
          mountPath: /hostfs/proc
          readOnly: true
        - name: cgroup
          mountPath: /hostfs/sys/fs/cgroup
          readOnly: true
---
# Source: metricbeat/charts/kube-state-metrics/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metricbeat-kube-state-metrics
  labels:
    app.kubernetes.io/name: kube-state-metrics
    helm.sh/chart: "kube-state-metrics-2.4.1"
    app.kubernetes.io/instance: "metricbeat"
    app.kubernetes.io/managed-by: "Tiller"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/instance: "metricbeat"
    spec:
      hostNetwork: false
      serviceAccountName: metricbeat-kube-state-metrics
      securityContext:
        fsGroup: 65534
        runAsUser: 65534
      containers:
      - name: kube-state-metrics
        args:

        - --collectors=certificatesigningrequests


        - --collectors=configmaps


        - --collectors=cronjobs


        - --collectors=daemonsets


        - --collectors=deployments


        - --collectors=endpoints


        - --collectors=horizontalpodautoscalers


        - --collectors=ingresses


        - --collectors=jobs


        - --collectors=limitranges


        - --collectors=namespaces


        - --collectors=nodes


        - --collectors=persistentvolumeclaims


        - --collectors=persistentvolumes


        - --collectors=poddisruptionbudgets


        - --collectors=pods


        - --collectors=replicasets


        - --collectors=replicationcontrollers


        - --collectors=resourcequotas


        - --collectors=secrets


        - --collectors=services


        - --collectors=statefulsets


        - --collectors=storageclasses



        imagePullPolicy: IfNotPresent
        image: "quay.io/coreos/kube-state-metrics:v1.8.0"
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 5
          timeoutSeconds: 5
---
# Source: metricbeat/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: 'metricbeat-metricbeat-metrics'
  labels:
    app: 'metricbeat-metricbeat-metrics'
    chart: 'metricbeat-7.5.2'
    heritage: 'Tiller'
    release: 'metricbeat'
spec:
  replicas: 1
  selector:
    matchLabels:
      app: 'metricbeat-metricbeat-metrics'
      chart: 'metricbeat-7.5.2'
      heritage: 'Tiller'
      release: 'metricbeat'
  template:
    metadata:
      annotations:
        
        configChecksum: f97bed025a57e28c0a7779e8af6bfc4da29d2dd20a660e480214637511beedc
      labels:
        app: 'metricbeat-metricbeat-metrics'
        chart: 'metricbeat-7.5.2'
        heritage: 'Tiller'
        release: 'metricbeat'
    spec:
      serviceAccountName: metricbeat-metricbeat
      terminationGracePeriodSeconds: 30
      volumes:
      - name: elastic-certificates
        secret:
          secretName: elastic-certificates
      - name: metricbeat-config
        configMap:
          defaultMode: 0600
          name: metricbeat-metricbeat-config
      imagePullSecrets:
        - name: gitlab-docker-registry-credential
        
      containers:
      - name: "metricbeat"
        image: "registry.gitlab.com/parebem/research/elastic-stack/infrastructure-docker-images/metricbeat:7.5.2-1"
        imagePullPolicy: "IfNotPresent"
        args:
          - "-c"
          - "/usr/share/metricbeat/kube-state-metrics-metricbeat.yml"
          - "-e"
          - "-E"
          - "http.enabled=true"
        livenessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              curl --fail 127.0.0.1:5066
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              metricbeat test output
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 5
          
        resources:
          limits:
            cpu: 1000m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
          
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: KUBE_STATE_METRICS_HOSTS
          value: "$(METRICBEAT_KUBE_STATE_METRICS_SERVICE_HOST):$(METRICBEAT_KUBE_STATE_METRICS_SERVICE_PORT_HTTP)"
        - name: ELASTICSEARCH_HOSTS
          value: https://elasticsearch-master.mobvaga-monitoring.svc.cluster.local:9200
        - name: ELASTICSEARCH_USERNAME
          valueFrom:
            secretKeyRef:
              key: username
              name: elastic-credentials
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: elastic-credentials
        
        securityContext:
          privileged: false
          runAsUser: 0
          
        volumeMounts:
        - name: elastic-certificates
          mountPath: /usr/share/metricbeat/config/certs
        - name: metricbeat-config
          mountPath: /usr/share/metricbeat/kube-state-metrics-metricbeat.yml
          readOnly: true
          subPath: kube-state-metrics-metricbeat.yml
        - name: metricbeat-config
          mountPath: /usr/share/metricbeat/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml

Describe the bug:
After apply all this helms I check the kibana metrics home page and check the hosts. The host dont was with the name of metric's pod. And when I check the host.name attribute sent by filebeat, the same bug, but now with the filebeat pod's name.

Steps to reproduce:

Run a kubernetes cluster (I believe the same happen on minikube)
Apply elastic helms (elasticsearch, kibana, metricbeat and filebeat)
Check the host page

Expected behavior:
Show the real hostname.

Any additional context:
After a extensive research to find the reason I found this a discussion on elastic site talking about that. In this discussion someone recommend to use hostNetwork as true.
I check it and it really works. But filebeat and metricbeat conflict on port 5066.

https://discuss.elastic.co/t/metricbeat-on-kubernetes-field-beat-hostname-got-the-pod-name-and-no-the-host-name/164438

The text was updated successfully, but these errors were encountered:

jmlrt · 2020-02-03T20:36:19Z

related to #355, #394 and #449

jmlrt · 2020-03-05T09:05:41Z

I'm closing it as duplicate of #355.

jmlrt added the duplicate This issue or pull request already exists label Feb 3, 2020

jmlrt closed this as completed Mar 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metricbeat and Filebeat dont send correct hostname (host.name) #462

Metricbeat and Filebeat dont send correct hostname (host.name) #462

talesporto commented Jan 27, 2020 •

edited by jmlrt

Loading

jmlrt commented Feb 3, 2020

jmlrt commented Mar 5, 2020

Metricbeat and Filebeat dont send correct hostname (host.name) #462

Metricbeat and Filebeat dont send correct hostname (host.name) #462

Comments

talesporto commented Jan 27, 2020 • edited by jmlrt Loading

jmlrt commented Feb 3, 2020

jmlrt commented Mar 5, 2020

talesporto commented Jan 27, 2020 •

edited by jmlrt

Loading