Align the version of Elasticsearch/Logstash/Kibana (7.16) across all integration tests and documentation #25663

yrodiere · 2022-05-18T16:08:59Z

In short, we need at least Elasticsearch 7.11 to run on Apple M1, and the heterogeneous configuration made upgrading the Elasticsearch version difficult.

This makes the Elasticsearch container configuration consistent across all integration tests and bumps the Elasticsearch server version used for tests to 7.16 (also Logstash and Kibana, so technically we're upgrading the whole ELK stack).

I didn't change the version of the client, in particular the High Level REST client, which cannot be upgraded beyond 7.10 because 7.11+ is no longer open-source. However, that's no big deal as the High Level REST client is forward-compatible with Elasticsearch (server) 7.16. We will have a problem when we want to upgrade to Elasticsearch 8.x, but that's a problem for another day.

…ration tests

…n of logging-gelf Because older versions don't support Elasticsearch 7.10. See https://archivedocs.graylog.org/en/latest/pages/configuration/elasticsearch.html See https://docs.graylog.org/docs/installing#system-requirements

…tion tests and documentation

…ration tests docker.elastic.co has been down a few times in the past, dockerhub is more reliable. They both serve the exact same images (same tags, same hashes), so the change should be mostly transparent.

1. Because we don't use it (we always connect unauthenticated) 2. Because we'll need to disable it in Elasticsearch 8+ anyway, otherwise Elasticsearch uses a self-signed certificate (which we have to allow in the client JVM) and mandates HTTP authentication.

quarkus-bot · 2022-05-19T16:56:52Z

Failing Jobs - Building `4076577`

Status	Name	Step	Failures	Logs	Raw logs
✖	Devtools Tests - JDK 11	`Build`	Failures	Logs	Raw logs
✖	Devtools Tests - JDK 17	`Build`	Failures	Logs	Raw logs
✖	Native Tests - HTTP	`Build`	Failures	Logs	Raw logs

Full information is available in the Build summary check run.

Failures

⚙️ Devtools Tests - JDK 11 #

- Failing: integration-tests/devtools

📦 integration-tests/devtools

✖ io.quarkus.platform.catalog.RegistrySnapshotCatalogCompatibilityTest.testRegistrySnapshotPlatformCatalog line 30 - More details - Source on GitHub

java.lang.IllegalStateException: Failed to locate the dev tools config file (.quarkus/config.yaml)
	at io.quarkus.devtools.testing.registry.client.TestRegistryClient.<init>(TestRegistryClient.java:33)
	at io.quarkus.devtools.testing.registry.client.TestRegistryClientFactory.buildRegistryClient(TestRegistryClientFactory.java:32)

⚙️ Devtools Tests - JDK 17 #

- Failing: integration-tests/devtools

📦 integration-tests/devtools

✖ io.quarkus.platform.catalog.RegistrySnapshotCatalogCompatibilityTest.testRegistrySnapshotPlatformCatalog line 30 - More details - Source on GitHub

java.lang.IllegalStateException: Failed to locate the dev tools config file (.quarkus/config.yaml)
	at io.quarkus.devtools.testing.registry.client.TestRegistryClient.<init>(TestRegistryClient.java:33)
	at io.quarkus.devtools.testing.registry.client.TestRegistryClientFactory.buildRegistryClient(TestRegistryClientFactory.java:32)

⚙️ Native Tests - HTTP #

- Failing: integration-tests/resteasy-reactive-kotlin/standard

📦 integration-tests/resteasy-reactive-kotlin/standard

✖ io.quarkus.it.resteasy.reactive.kotlin.NativeReactiveGreetingResourceIT.testHello - More details - Source on GitHub

java.lang.RuntimeException: java.util.concurrent.CompletionException: java.lang.RuntimeException: Unable to start Quarkus test resource io.quarkus.it.resteasy.reactive.kotlin.KafkaTestResource@3f376582
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.throwBootFailureException(QuarkusIntegrationTestExtension.java:284)
	at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeEach(QuarkusIntegrationTestExtension.java:76)

gsmet · 2022-05-19T18:35:33Z

@quarkusio/devtools the RegistrySnapshotCatalogCompatibilityTest.testRegistrySnapshotPlatformCatalog test is failing for several PRs, could one of you have a look or disable it?

gastaldi · 2022-05-19T18:41:24Z

@gsmet weird, it's disabled already: https://github.com/quarkusio/quarkus/blob/main/integration-tests/devtools/src/test/java/io/quarkus/platform/catalog/RegistrySnapshotCatalogCompatibilityTest.java#L36

gsmet · 2022-05-21T21:17:47Z

@gastaldi AFAICS from the bot's report, it's this thing throwing the exception:
https://github.com/quarkusio/quarkus/blob/main/integration-tests/devtools/src/test/java/io/quarkus/platform/catalog/RegistrySnapshotCatalogCompatibilityTest.java#L30

gsmet · 2022-05-21T21:19:37Z

docs/src/main/asciidoc/centralized-log-management.adoc

@@ -103,11 +102,12 @@ version: '3.2'

 services:
  elasticsearch:
-    image: docker.elastic.co/elasticsearch/elasticsearch-oss:{es-version}
+    image: {elasticsearch-image}


We need to include this attribute here:
https://github.com/quarkusio/quarkusio.github.io/blob/develop/_guides/attributes.adoc

Let's do it now so that we don't forget about it.

Here you go: quarkusio/quarkusio.github.io#1438

I don't really know what I'm doing though; I would have expected this file to be generated. Should I remember to keep it in sync every time we upgrade to a new version of Elasticsearch?

Yeah unfortunately it's not generated. It wasn't when I took over the role and I never took the time to think about how to do this properly.

@gsmet

As requested by @gsmet [here](quarkusio/quarkus#25663 (review)). See https://github.com/quarkusio/quarkus/pull/25663/files#diff-749f6f9897e14f3c05890960c13ad402b7541d714116175a61cada4e64f085a1R2827-R2829 See https://github.com/quarkusio/quarkus/pull/25663/files#diff-150bdf617c2dc80dc5a9d5f1313f44a73950a96ca7b166b79a35b15f4c84f193R79-R82

@gsmet

As requested by @gsmet [here](quarkusio/quarkus#25663 (review)). See https://github.com/quarkusio/quarkus/pull/25663/files#diff-749f6f9897e14f3c05890960c13ad402b7541d714116175a61cada4e64f085a1R2827-R2829 See https://github.com/quarkusio/quarkus/pull/25663/files#diff-150bdf617c2dc80dc5a9d5f1313f44a73950a96ca7b166b79a35b15f4c84f193R79-R82

Fix cgroups configuration of docker volume used in logging-gelf integ…

f0c38b9

…ration tests

quarkus-bot bot added area/dependencies Pull requests that update a dependency file area/documentation area/elasticsearch area/hibernate-search Hibernate Search area/logging labels May 18, 2022

yrodiere mentioned this pull request May 18, 2022

Updates to docker-maven-plugin configuration so M1 mac build runs (almost) clean with -Dstart-containers #25648

Closed

yrodiere force-pushed the elasticsearch-it-image-consistency branch from e4da5a3 to c77f8a7 Compare May 18, 2022 16:32

This comment has been minimized.

Sign in to view

yrodiere added 4 commits May 19, 2022 09:29

Fix outdated properties in logging-gelf IT readme

af0e8cc

Align the version of Elasticsearch/Logstash/Kibana across all integra…

4910216

…tion tests and documentation

Upgrade to ELK 7.16.3 for tests and documentation

a032e30

yrodiere force-pushed the elasticsearch-it-image-consistency branch from c77f8a7 to 79a45eb Compare May 19, 2022 07:29

This comment has been minimized.

Sign in to view

yrodiere added 2 commits May 19, 2022 14:10

Switch to dockerhub for Elasticsearch/Logstash/Kibana images in integ…

22c12c0

…ration tests docker.elastic.co has been down a few times in the past, dockerhub is more reliable. They both serve the exact same images (same tags, same hashes), so the change should be mostly transparent.

yrodiere force-pushed the elasticsearch-it-image-consistency branch from 79a45eb to 4076577 Compare May 19, 2022 12:10

gsmet reviewed May 21, 2022

View reviewed changes

gsmet approved these changes May 21, 2022

View reviewed changes

gsmet merged commit 263941c into quarkusio:main May 21, 2022

quarkus-bot bot added this to the 2.10 - main milestone May 21, 2022

yrodiere mentioned this pull request May 23, 2022

Add attributes for Elasticsearch/Logstash/Kibana images quarkusio/quarkusio.github.io#1438

Merged

yrodiere deleted the elasticsearch-it-image-consistency branch May 31, 2022 09:35

holly-cummins mentioned this pull request Jun 23, 2022

Use variable for volume modifier because :Z does not play nicely on M1 #26327

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Align the version of Elasticsearch/Logstash/Kibana (7.16) across all integration tests and documentation #25663

Align the version of Elasticsearch/Logstash/Kibana (7.16) across all integration tests and documentation #25663

yrodiere commented May 18, 2022

This comment has been minimized.

This comment has been minimized.

quarkus-bot bot commented May 19, 2022

gsmet commented May 19, 2022

gastaldi commented May 19, 2022

gsmet commented May 21, 2022

gsmet May 21, 2022

yrodiere May 23, 2022

gsmet May 23, 2022

Align the version of Elasticsearch/Logstash/Kibana (7.16) across all integration tests and documentation #25663

Align the version of Elasticsearch/Logstash/Kibana (7.16) across all integration tests and documentation #25663

Conversation

yrodiere commented May 18, 2022

This comment has been minimized.

This comment has been minimized.

quarkus-bot bot commented May 19, 2022

Failing Jobs - Building 4076577

Failures

⚙️ Devtools Tests - JDK 11 #

📦 integration-tests/devtools

⚙️ Devtools Tests - JDK 17 #

📦 integration-tests/devtools

⚙️ Native Tests - HTTP #

📦 integration-tests/resteasy-reactive-kotlin/standard

gsmet commented May 19, 2022

gastaldi commented May 19, 2022

gsmet commented May 21, 2022

gsmet May 21, 2022

Choose a reason for hiding this comment

yrodiere May 23, 2022

Choose a reason for hiding this comment

gsmet May 23, 2022

Choose a reason for hiding this comment

Failing Jobs - Building `4076577`