kibana docker container won't start

[graphaelli]

Kibana version: 8.0.0-SNAPSHOT

Original install method (e.g. download page, yum, from source, etc.): docker

Describe the bug:

Kibana container won't start with default config

Steps to reproduce:

1. docker run --rm -it docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT

  log   [23:17:42.582] [fatal][root] Error: Invalid server options  {
  "port": 5601,
  "routes": {
    "cache": {
      "privacy": "private",
      "otherwise": "private, no-cache, no-store, must-revalidate"
    },
    "cors": false,
    "payload": {
      "maxBytes": 1048576
    },
    "validate": {
      "failAction": function defaultValidationErrorHandler(request, h, err) {\n  // Newer versions of Joi don't format the key for missing params the same way. This shim\n  // provides backwards compatibility. Unfortunately, Joi doesn't export it's own Error class\n  // in JS so we have to rely on the `name` key before we can cast it.\n  //\n  // The Hapi code we're 'overwriting' can be found here:\n  //     https://github.com/hapijs/hapi/blob/master/lib/validation.js#L102\n  if (err && err.name === 'ValidationError' && err.hasOwnProperty('output')) {\n    const validationError = err;\n    const validationKeys = [];\n    validationError.details.forEach(detail => {\n      if (detail.path.length > 0) {\n        validationKeys.push(_hoek.default.escapeHtml(detail.path.join('.')));\n      } else {\n        // If no path, use the value sigil to signal the entire value had an issue.\n        validationKeys.push('value');\n      }\n    });\n    validationError.output.payload.validation.keys = validationKeys;\n  }\n\n  throw err;\n},
      "options": {
        "abortEarly": false
      },
      "payload": "[value => Promise.resolve((0, _prototype_pollution.validateObject)(value))]"
    }
  },
  "state": {
    "strictHeader": false,
    "isHttpOnly": true,
    "isSameSite": false
  },
  "host" [1]: "0"
}

[1] "host" must be a valid hostname
    at Object.exports.apply (/usr/share/kibana/node_modules/@hapi/hapi/lib/config.js:19:15)
    at Object.internals.setup (/usr/share/kibana/node_modules/@hapi/hapi/lib/core.js:649:23)
    at new module.exports.internals.Core (/usr/share/kibana/node_modules/@hapi/hapi/lib/core.js:116:46)
    at new module.exports (/usr/share/kibana/node_modules/@hapi/hapi/lib/server.js:23:18)
    at createServer (/usr/share/kibana/src/core/server/http/http_tools.js:119:18)
    at HttpServer.setup (/usr/share/kibana/src/core/server/http/http_server.js:86:48)
    at HttpService.runNotReadyServer (/usr/share/kibana/src/core/server/http/http_service.js:165:26)
    at HttpService.setup (/usr/share/kibana/src/core/server/http/http_service.js:80:18)
    at Server.setup (/usr/share/kibana/src/core/server/server.js:192:23)
    at Root.setup (/usr/share/kibana/src/core/server/root/index.js:47:14)
    at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:110:5)
    at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:180:5)

 FATAL  Error: Invalid server options  {
  "port": 5601,
  "routes": {
    "cache": {
      "privacy": "private",
      "otherwise": "private, no-cache, no-store, must-revalidate"
    },
    "cors": false,
    "payload": {
      "maxBytes": 1048576
    },
    "validate": {
      "failAction": function defaultValidationErrorHandler(request, h, err) {\n  // Newer versions of Joi don't format the key for missing params the same way. This shim\n  // provides backwards compatibility. Unfortunately, Joi doesn't export it's own Error class\n  // in JS so we have to rely on the `name` key before we can cast it.\n  //\n  // The Hapi code we're 'overwriting' can be found here:\n  //     https://github.com/hapijs/hapi/blob/master/lib/validation.js#L102\n  if (err && err.name === 'ValidationError' && err.hasOwnProperty('output')) {\n    const validationError = err;\n    const validationKeys = [];\n    validationError.details.forEach(detail => {\n      if (detail.path.length > 0) {\n        validationKeys.push(_hoek.default.escapeHtml(detail.path.join('.')));\n      } else {\n        // If no path, use the value sigil to signal the entire value had an issue.\n        validationKeys.push('value');\n      }\n    });\n    validationError.output.payload.validation.keys = validationKeys;\n  }\n\n  throw err;\n},
      "options": {
        "abortEarly": false
      },
      "payload": "[value => Promise.resolve((0, _prototype_pollution.validateObject)(value))]"
    }
  },
  "state": {
    "strictHeader": false,
    "isHttpOnly": true,
    "isSameSite": false
  },
  "host" [1]: "0"
}

[1] "host" must be a valid hostname

Expected behavior: container starts like it used to

Any additional context:

Maybe #85406 introduced the issue? Just a guess right now.

Working around this with by setting SERVER_HOST env with:

docker run --rm -it -e SERVER_HOST=0.0.0.0 docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT 

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[watson]

This is related to the hapi v20 upgrade (#85406). Not sure exactly what changed yet...

[watson]

The root cause is that the validator for the host option that hapi is being configured with changed from this regex (defined in @hapi/joi):

/^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$/

Which allows the string 0 - to the method isDomainValid defined in @hapi/validate:

https://github.com/hapijs/validate/blob/b2d63af83715f5641e6b5c683d9a79952c445286/lib/types/string.js#L677-L721

Which doesn't accept the string 0 as a valid host.

[watson]

Dug a little deeper: hapi pre-v20 will pass on the 0 host to the node-core server.listen() function (because it doesn't throw a validation error), which internally will try to look up the DNS for the host 0, which resolves to the IP 0.0.0.0 (i.e. loopback), which it then binds the socket to. This is why it works.

(Internally dns.lookup runs this C++ function: https://github.com/nodejs/node/blob/e725355e5cdd77ac9287dea9c0d2958c994e8c27/src/cares_wrap.cc#L1940-L1997)

[watson]

My recommendation is to update the configuration to use server.host: "0.0.0.0" if we can - unless users rely on using 0?

[watson]

If I understand correctly, then elastic-package is used to generate the Docker packages? If so, I think this is the line we need to change: https://github.com/elastic/elastic-package/blob/fd6ce022238df2cb134c91d726e0225f500cfc54/internal/install/static_kibana_config_yml.go#L9

[graphaelli]

I'm not sure if that's addressed to me but I'm not using elastic-package.

Since this was backported to 7.x I think this a breaking change and should be handled by kibana itself. server.host: 0 is also shipped in the default kibana docker container image and is mentioned in docs - all of those would need an update too.

[watson]

I created a PR to fix this by converting server.host value 0 to 0.0.0.0: #86806