Kibana YAML settings with empty array values get parsed as null

[an-tex]

Kibana version: 7.14.1

Elasticsearch version: 7.14.1

ECK version: 1.7.1

Steps to reproduce:

1. Setup elasticsearch using ECK
2. Create kibana using:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic-monitoring
spec:
  config:
    xpack.fleet:
      agentPolicies:
      - id: 1
        is_default_fleet_server: true
        is_managed: true
        # this is the problematic field
        monitoring_enabled: []
        name: Default Fleet Server on ECK policy
        package_policies:
        - name: fleet_server-1
          package:
            name: fleet_server
      agents.elasticsearch.hosts:
      - https://elasticsearch-es-http:9200
      agents.fleet_server.hosts:
      - https://fleet-server-agent-http:8220
      packages:
      - name: apm
        version: latest
  count: 2
  elasticsearchRef:
    name: elasticsearch
  version: 7.14.1

3. Kibana won't start due to:

{"type":"log","@timestamp":"2021-09-06T12:30:16+00:00","tags":["fatal","root"[],"pid":1209,"message":"ValidationError: [config validation of [xpack.fleet[].agentPolicies.0.monitoring_enabled]: expected value of type [array[] but got [null[]\n    at ObjectType.validate (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:88:13)\n    at ConfigService.validateAtPath (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:227:19)\n    at MapSubscriber.project (/usr/share/kibana/node_modules/@kbn/config/target_node/config_service.js:235:169)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/kibana/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/kibana/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:44:25)\n    at ReplaySubject.Subject._trySubscribe (/usr/share/kibana/node_modules/rxjs/internal/Subject.js:102:51)\n    at ReplaySubject.Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:30:22)\n    at MapSubscriber.shareReplayOperation (/usr/share/kibana/node_modules/rxjs/internal/operators/shareReplay.js:48:32)\n    at Observable.subscribe (/usr/share/kibana/node_modules/rxjs/internal/Observable.js:25:31)\n    at MapOperator.call (/usr/share/kibana/node_modules/rxjs/internal/operators/map.js:32:23) {\n  cause: SchemaTypeError: expected value of type [array[] but got [null[]\n      at ArrayType.onError (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:130:14)\n      at Object.error (/usr/share/kibana/node_modules/@kbn/config-schema/target_node/types/type.js:72:47)\n      at Object.internals.finalize (/usr/share/kibana/node_modules/joi/lib/validator.js:472:36)\n      at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:239:34)\n      at internals.Base.$_validate (/usr/share/kibana/node_modules/joi/lib/base.js:773:26)\n      at Object.validate (/usr/share/kibana/node_modules/joi/lib/types/keys.js:108:45)\n      at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:325:26)\n      at internals.Base.$_validate (/usr/share/kibana/node_modules/joi/lib/base.js:773:26)\n      at Object.validate (/usr/share/kibana/node_modules/joi/lib/types/array.js:273:45)\n      at Object.internals.rules (/usr/share/kibana/node_modules/joi/lib/validator.js:394:33)\n      at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:350:22)\n      at internals.Base.$_validate (/usr/share/kibana/node_modules/joi/lib/base.js:773:26)\n      at Object.validate (/usr/share/kibana/node_modules/joi/lib/types/keys.js:108:45)\n      at Object.exports.validate (/usr/share/kibana/node_modules/joi/lib/validator.js:325:26)\n      at Object.internals.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:142:28)\n      at Object.exports.entry (/usr/share/kibana/node_modules/joi/lib/validator.js:27:30) {\n    cause: undefined,\n    path: [ 'agentPolicies', '0', 'monitoring_enabled' ]\n  }\n}"}
 FATAL  Error: [config validation of [xpack.fleet[].agentPolicies.0.monitoring_enabled]: expected value of type [array] but got [null]

As of elastic/kibana#97328 and elastic/kibana#97356 this should be possible

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jen-huang]

This issue is specific to the ECK environment. It is the same issue as #3718, where a workaround was implemented but only for another specific setting.

[pebrc]

Just to refresh the context as to why we lose the empty array through the various layers of parsing:

• API server: []
• ECK Kibana controller
  • (after un-marshalling the CRD): nil(map[string]interface{})
  • NewCanonicalConfigFrom: yaml.Marshal -> []bytes -> yaml.Unmarshal -> nil(map[string]interface{}) -> ucfg.Config`
  • Render(): ucf.Config.Unpack(map[string]interface{}) -> yaml.Marshal

I think the Render() is key here: we are unpacking the configuration into an map[string]interface{} because we don't model the Kibana configuration strictly. It is just a bag of JSON or YAML in this case. However that means that we have no type information about what that nil value means (we already lost that on the first unmarshalling I believe). So we are rendering a nil valued type interface{} which turns out to be null in the resulting YAML.

[thbkrkr]

Verified, this is fixed by elastic/go-ucfg#188 included in ECK via #5037.