[BUG] - [upgrading 0.4 to 0.5/0.6]: Elasticsearch service failing on RedHat clusters after upgrading

[przemyslavic]

Describe the bug
Elasticsearch service failing on RedHat clusters after upgrading

To Reproduce
Steps to reproduce the behavior:
TBD

Expected behavior
Successfully upgraded ES.

Config files
TDB

OS (please complete the following information):

• RHEL

Cloud Environment (please complete the following information):

• Azure
• AWS

Additional context

[2020-04-29T07:27:27,622][INFO ][o.e.e.NodeEnvironment    ] [ci-04azurrhelflannel-logging-vm-0] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [28.4gb], net total_space [31.4gb], types [rootfs]
[2020-04-29T07:27:27,672][INFO ][o.e.e.NodeEnvironment    ] [ci-04azurrhelflannel-logging-vm-0] heap size [1015.6mb], compressed ordinary object pointers [true]
[2020-04-29T07:27:28,140][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] node name [ci-04azurrhelflannel-logging-vm-0], node ID [ODNQa03tTPeti00VhzwM0A], cluster name [elasticsearch]
[2020-04-29T07:27:28,141][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] version[7.6.2], pid[4442], build[oss/rpm/ef48eb35cf30adf4db14086e8aabd07ef6fb113f/2020-03-26T06:34:37.794943Z], OS[Linux/3.10.0-957.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.2/13.0.2+8]
[2020-04-29T07:27:28,141][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] JVM home [/usr/share/elasticsearch/jdk]
[2020-04-29T07:27:28,141][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-10422194386497553304, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=536870912, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2020-04-29T07:27:31,330][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [aggs-matrix-stats]
[2020-04-29T07:27:31,331][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [analysis-common]
[2020-04-29T07:27:31,331][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [ingest-common]
[2020-04-29T07:27:31,331][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [ingest-geoip]
[2020-04-29T07:27:31,331][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [ingest-user-agent]
[2020-04-29T07:27:31,331][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [lang-expression]
[2020-04-29T07:27:31,332][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [lang-mustache]
[2020-04-29T07:27:31,332][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [lang-painless]
[2020-04-29T07:27:31,332][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [mapper-extras]
[2020-04-29T07:27:31,332][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [parent-join]
[2020-04-29T07:27:31,332][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [percolator]
[2020-04-29T07:27:31,333][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [rank-eval]
[2020-04-29T07:27:31,333][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [reindex]
[2020-04-29T07:27:31,333][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [repository-url]
[2020-04-29T07:27:31,333][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [systemd]
[2020-04-29T07:27:31,333][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] loaded module [transport-netty4]
[2020-04-29T07:27:31,334][INFO ][o.e.p.PluginsService     ] [ci-04azurrhelflannel-logging-vm-0] no plugins loaded
[2020-04-29T07:27:38,574][INFO ][o.e.d.DiscoveryModule    ] [ci-04azurrhelflannel-logging-vm-0] using discovery type [zen] and seed hosts providers [settings]
[2020-04-29T07:27:40,054][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] initialized
[2020-04-29T07:27:40,055][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] starting ...
[2020-04-29T07:27:40,513][INFO ][o.e.t.TransportService   ] [ci-04azurrhelflannel-logging-vm-0] publish_address {10.1.3.4:9300}, bound_addresses {10.1.3.4:9300}
[2020-04-29T07:27:41,464][INFO ][o.e.b.BootstrapChecks    ] [ci-04azurrhelflannel-logging-vm-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-04-29T07:27:41,490][ERROR][o.e.b.Bootstrap          ] [ci-04azurrhelflannel-logging-vm-0] node validation exception
[1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-04-29T07:27:41,598][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] stopping ...
[2020-04-29T07:27:41,736][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] stopped
[2020-04-29T07:27:41,736][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] closing ...
[2020-04-29T07:27:41,815][INFO ][o.e.n.Node               ] [ci-04azurrhelflannel-logging-vm-0] closed

[rafzei]

@przemyslavic or @to-bar is this still valid? Could you provide more details like:

• is AWS, other cni plugins also affected?
• it appeared only with one instance of logging vm?

[mkyc]

Looks like duplicate of #922

[ghost]

Yup, looks like duplicate. We need to get answers for Rafal comments first, and then we can start the investigation.

[to-bar]

The root cause of this issue is that in epirepo we have elasticsearch-oss package in version 6.x.y and 7.v.z as well and the latter obsoletes the former, i.e. yum "implicitly" installs 7.v.z instead of 6.x.y.

[to-bar]

Duplicate of #922