[RHEL/CentOS] Elasticsearch v6 stops working after epicli upgrade

[przemyslavic]

Describe the bug
There is an issue with Elasticsearch service not starting on RHEL environments after cluster upgrading.

To Reproduce
Steps to reproduce the behavior:

1. Deploy a 0.4.4 cluster (with logging vm)
2. Run epicli upgrade

Expected behavior
The cluster has been upgraded successfully. All components are working properly

Config files

...
    logging:
      count: 1
      machine: logging-machine-rhel
...
---
kind: infrastructure/virtual-machine
name: logging-machine-rhel
provider: azure
based_on: logging-machine
specification:
  storage_image_reference:
    publisher: RedHat
    offer: RHEL
    sku: 7-RAW
    version: "7.7.2019090418"

OS (please complete the following information):

• OS: [RHEL]

Cloud Environment (please complete the following information):

• Cloud Provider [all]

Additional context

After upgrading there is elasticsearch 7.6.0 instead of 6.8.5 installed on RHEL vms.

Installed Packages
elasticsearch-oss.x86_64 7.6.0-1 @epirepo

[root@vm-ci-upgrades-rhel-elk-001 bin]# ./elasticsearch --version
Version: 7.6.0, Build: oss/rpm/2020-02-06T00:09:00.449973Z, JVM: 13.0.2

The service does not start

[2020-02-17T12:28:05,115][INFO ][o.e.d.DiscoveryModule ] [vm-ci-upgrades-rhel-elk-001] using discovery type [zen] and seed hosts providers [settings]
[2020-02-17T12:28:07,040][INFO ][o.e.n.Node ] [vm-ci-upgrades-rhel-elk-001] initialized
[2020-02-17T12:28:07,040][INFO ][o.e.n.Node ] [vm-ci-upgrades-rhel-elk-001] starting ...
[2020-02-17T12:28:07,555][INFO ][o.e.t.TransportService ] [vm-ci-upgrades-rhel-elk-001] publish_address {10.0.0.5:9300}, bound_addresses {10.0.0.5:9300}
[2020-02-17T12:28:08,701][INFO ][o.e.b.BootstrapChecks ] [vm-ci-upgrades-rhel-elk-001] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-02-17T12:28:08,704][ERROR][o.e.b.Bootstrap ] [vm-ci-upgrades-rhel-elk-001] node validation exception
[1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-02-17T12:28:08,818][INFO ][o.e.n.Node ] [vm-ci-upgrades-rhel-elk-001] stopping ...
[2020-02-17T12:28:08,950][INFO ][o.e.n.Node ] [vm-ci-upgrades-rhel-elk-001] stopped
[2020-02-17T12:28:08,951][INFO ][o.e.n.Node ] [vm-ci-upgrades-rhel-elk-001] closing ...

We also need to update the upgrade.yml file by removing the when condition:

- hosts: elasticsearch
  become: true
  become_method: sudo
  tasks:
    - import_role:
        name: upgrade
        tasks_from: elasticsearch
      when: ansible_os_family == 'Debian' # TODO: Fix issue #922 for RedHat

[mkyc]

@przemyslavic can you groom this a bit?

[przemyslavic]

@mkyc just updated.

[to-bar]

We need to fix this issue before releasing 0.5.5, 0.6.1 and 0.7.2.

[mkyc]

@to-bar can you create backport issues? We should not solve all those with one issue.

[to-bar]

Test OK (upgrade from 0.4.5 to develop)