Failure to appy schema and metricbeat.sock not found errors in Fleet Server metricbeat logs

[david-kow]

When running Fleet Server in Kubernetes with agent monitoring enabled I see the following three errors repeat continuously in the metricbeat logs:

{"log.level":"info","@timestamp":"2021-07-29T08:49:10.756Z","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: error making http request: Get \"http://unix/state\": dial unix /usr/share/elastic-agent/state/data/tmp/default/metricbeat/metricbeat.sock: connect: no such file or directory","service.name":"metricbeat","event.dataset":"metricbeat_monitor-json.log","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-07-29T08:49:10.848Z","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.state: failure to apply state schema: 4 errors: key `management` not found; key `module` not found; key `output` not found; key `queue` not found","service.name":"metricbeat","event.dataset":"metricbeat_monitor-json.log","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-07-29T08:49:10.851Z","log.origin":{"file.name":"module/wrapper.go","file.line":259},"message":"Error fetching data for metricset beat.stats: failure to apply stats schema: 1 error: key `libbeat` not found","service.name":"metricbeat","event.dataset":"metricbeat_monitor-json.log","ecs.version":"1.6.0"}

• Version: 7.14.0-SNAPSHOT
• Operating System: centos
• fleet-setup.yml contents:

# cat fleet-setup.yml
fleet:
  ca: /usr/share/fleet-server/config/http-certs/ca.crt
  enroll: true
  url: https://fleet-server-agent-http.default.svc:8220
fleet_server:
  cert: /usr/share/fleet-server/config/http-certs/tls.crt
  cert_key: /usr/share/fleet-server/config/http-certs/tls.key
  elasticsearch:
    ca: /mnt/elastic-internal/elasticsearch-association/default/elasticsearch/certs/ca.crt
    host: https://elasticsearch-es-http.default.svc:9200
    password: redacted
    username: default-fleet-server-default-elasticsearch-agent-user
  enable: true
kibana:
  fleet:
    ca: /mnt/elastic-internal/kibana-association/default/kibana/certs/ca.crt
    host: https://kibana-kb-http.default.svc:5601
    password: redacted
    setup: true
    username: default-fleet-server-agent-kb-user

• Fleet Server is visible in the UI:

• Errors are visible in the Discover:

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[david-kow]

Below the config from "View policy".

id: c41271e0-f05d-11eb-943d-d9ccf88ccf0a
revision: 1
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://elasticsearch-es-http.default.svc:9200'
output_permissions:
  default:
    _elastic_agent_checks:
      cluster:
        - monitor
      indices:
        - names:
            - logs-elastic_agent.*-default
            - metrics-elastic_agent.*-default
          privileges:
            - auto_configure
            - create_doc
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: 84e5207f-cec4-4156-a853-e48b88042987
    name: fleet_server-1
    revision: 1
    type: fleet-server
    use_output: default
    meta:
      package:
        name: fleet_server
        version: 0.9.1
    data_stream:
      namespace: default
    server:
      port: 8220
      host: 0.0.0.0
fleet:
  hosts:
    - 'https://fleet-server-agent-http.default.svc:8220'

[michalpristas]

is this message reoccurring or one time thing. are you able to extract elastic agent directory with logs, archive it and share?

[michalpristas]

no worries I have repro steps