Kibana 7.17.0 is not up and working with ES 8.0.0 after restart.

[bhavyarm]

Kibana version: 7.17 BC1

Elasticsearch version: 8.0.0 RC1

Server OS version: darwin_x86_64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): from staging

Describe the bug: Kibana 7.17 is not functional with ES 8.0.0.

Steps to reproduce:

1. Run ES 7.17.0/Kibana 7.17.0
2. Make sure you are ready for upgrade after going through all the steps in upgrade assistant
3. Download 8.0.0 ES/Kibana
4. Copy data and config folder of ES 7.17 into ES 8.0
5. Stop ES 7.17 and start ES 8.0
6. Kibana logs error but is still up

 log   [13:19:44.516] [error][elasticsearch-service] This version of Kibana (v7.17.0) is incompatible with the following Elasticsearch nodes in your cluster: v8.0.0 @ 127.0.0.1:9200 (127.0.0.1)

7. Restart Kibana
8. Kibana fails to come up and logs the same version incompatibility error

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

As already discussed, the fix itself it quite easy. The main issue is that we don't have ANY kind of automated testing around Kib/ES version mismatch in our FTR suites.

Technically, all we have to do to support 7.17 Kib -> 8.X ES is the same thing that was done in https://github.com/elastic/kibana/pull/30390/files

  // Accept the next major version of ES.
  if (esVersionNumbers.major === kibanaVersionNumbers.major + 1) {
    return true;
  }

The version check logic is here:

kibana/src/core/server/elasticsearch/version_check/es_kibana_version_compatability.ts

Lines 29 to 32 in 4584a8b

	// Reject mismatching major version numbers.
	if (esVersionNumbers.major !== kibanaVersionNumbers.major) {
	return false;
	}

[pgayvallet]

My main question now is: Do we want to support Kibana 7.17 connecting to ANY minor of ES, or do we want to only allow connecting to ES 8.0? Elasticsearch currently allows upgrading from 7.last to ANY 8.x version, so I feel like we don't have much choice but to align?

[mfinkle]

Elasticsearch currently allows upgrading from 7.last to ANY 8.x version, so I feel like we don't have much choice but to align?

I think it would be good to align. Pierre is right that we need to expand some of our FTR upgrade testing to make sure we have some test coverage.

[tylersmalley]

@pgayvallet, I pinged the stack upgrade team and got confirmation from @mfinkle that we "should assume that 7.last will upgrade to 8.0 and 8.x". Hopefully that helps move things forward.

[lukeelmers]

Okay, after further discussion with @tylersmalley @pgayvallet @mfinkle @LeeDr & @elasticdog, here is where we ended up:

• In a perfect world, we have an Elasticsearch API that Kibana can check for compatibility instead of having hard-coded version numbers. But there isn’t enough time to get such a solution introduced in ES for use in 8.0.
• Not knowing how our versioning strategy may evolve in the future, we are wary of either removing the check entirely, or pinning it to a specific version later in the 8.x series.
• So in the meantime, we plan to align with ES, i.e. allow Kibana upgrades from 7.17 to any 8.x minor, and then if it becomes clear we need to stop supporting upgrades at 8.whatever, we will weigh our options with the ES team and can take whatever actions are necessary at that time.

TLDR: for this particular task, we just need to update the version check logic as Pierre describes in #122459 (comment)

[pgayvallet]

Closed by #122627