Test fixtures krb5 #40297

alpar-t · 2019-03-21T09:05:46Z

Replaces the vagrant based kerberos fixtures with docker based test fixtures plugin.
The configuration is now entirely static on the docker side and no longer driven by Gradle,
also two different services are being configured since there are two different consumers of the fixture that can run in parallel and require different configurations.

@jbaiera the checksum error was due to the fact that both the local and the mapped ports need to be in the config since depending on where this is called upon one or the other will be used.

Closes #34095

Add two separate fixtures for the different configs used by this fixture.

This breaks on windows where TMP dir default to C:\Windows and startup fails with a permission error. I tried to create a tmp dir and pass in `TMP` env, but it lead to a class not found error, and since testclusers is already independent of the calling environment I stopped there.

We only ever connect on localhost. This entry just causes a missleading connection error

turns out we need both local and mapped port in the file

We need to disable these too when docker is not supproted.

elasticmachine · 2019-03-21T09:05:48Z

Pinging @elastic/es-core-infra

alpar-t · 2019-03-21T14:47:02Z

@elasticmachine test this please

mark-vieira · 2019-03-21T15:29:50Z

plugins/repository-hdfs/build.gradle


 // Create HDFS File System Testing Fixtures for HA/Secure combinations
 for (String fixtureName : ['hdfsFixture', 'haHdfsFixture', 'secureHdfsFixture', 'secureHaHdfsFixture']) {
  project.tasks.create(fixtureName, org.elasticsearch.gradle.test.AntFixture) {
-    dependsOn project.configurations.hdfsFixture
+    dependsOn project.configurations.hdfsFixture, project(':test:fixtures:krb5kdc-fixture').tasks.postProcessFixture


We need a better abstraction for this. This doesn't need to be done in this PR but something as simple as "this task needs this fixture to be done being setup" seems a common enough pattern to deserve it's own DSL.

I agree. We already add this dependency to testing tasks (RandomizedTestingTask that is, we'll have to add Gradle Test too there ).
This is the first example of a task in the "client" project other than a test needing to have the fixture up. I was thinking of creating a postProcessFixture task on the "client" project identical to the one on the fixture to allow for dumping ports to files, and depending on the fixture. What do you think ?

I was envisioning something similar to what we do for the rest tests. That is, we decorate tasks on the consuming project with some DSL which folks can use to indicate that task requires a given test fixture.

plugins/repository-hdfs/build.gradle

mark-vieira · 2019-03-21T15:37:23Z

plugins/repository-hdfs/build.gradle

+// Set the keytab files in the classpath so that we can access them from test code without the security manager
+// freaking out.
+project.dependencies {
+  testRuntime fileTree(dir: project(':test:fixtures:krb5kdc-fixture').ext.krb5Keytabs("hdfs","hdfs_hdfs.build.elastic.co.keytab").parent, include: ['*.keytab'])


FWIW, this kind of cross-project coupling makes me very uneasy. It's just so easy to break this kind of thing. We really should model these kinds of things as proper project dependencies. At the very least then we can ask Gradle "who uses this thing?".

We don't need to address this as part of the PR. Similar to my comment above, I think we need to provide better abstractions around things a fixture exposes to a dependent project.

test/fixtures/hdfs-fixture/build.gradle

x-pack/qa/kerberos-tests/build.gradle

mark-vieira

Changes LGTM 👍

alpar-t · 2019-03-25T16:05:25Z

Closes #39977 since the fixture is no longer vagrant based.

jbaiera

Just a quick sanity question, everything else LGTM

jbaiera · 2019-03-25T17:28:02Z

test/fixtures/krb5kdc-fixture/src/main/resources/provision/addprinc.sh

@@ -19,6 +19,9 @@

 set -e

+krb5kdc
+kadmind


Should these be in the hdfs.sh instead? Do these commands have any issues with being executed multiple times when adding multiple principals for a fixture?

To clarify "in hdfs.sh": I mean "in provision/hdfs.sh and provision/peppa.sh"

I tried it but the containers don't come online. since the tests work as-si I didn't look further at it, we can perhaps address this lather on.

alpar-t · 2019-03-26T06:01:10Z

@elasticmachine run elasticsearch-ci/2

alpar-t · 2019-03-27T07:03:27Z

@elasticmachine run elasticsearch-ci/1

alpar-t · 2019-03-27T07:04:17Z

@elasticmachine elasticsearch-ci/2

alpar-t · 2019-03-27T07:04:55Z

@elasticmachine run elasticsearch-ci/2

alpar-t · 2019-03-28T12:37:28Z

@elasticmachine run elasticsearch-ci/2

Replaces the vagrant based kerberos fixtures with docker based test fixtures plugin. The configuration is now entirely static on the docker side and no longer driven by Gradle, also two different services are being configured since there are two different consumers of the fixture that can run in parallel and require different configurations.

* master: (25 commits) [DOCS] Correct keystore commands for Email and Jira actions in Watcher (elastic#40417) [DOCS] Document common settings for snapshot repository plugins (elastic#40475) Remove with(out)-system-key tests (elastic#40547) Geo Point parse error fix (elastic#40447) Handle null retention leases in WaitForNoFollowersStep (elastic#40477) [DOCS] Adds anchors for ruby client (elastic#39867) Mute DataFrameAuditorIT#testAuditorWritesAudits Disable integTest when Docker is not available (elastic#40585) Add randomScore function in script_score query (elastic#40186) Test fixtures krb5 (elastic#40297) Correct ILM metadata minimum compatibility version (elastic#40569) SQL: Centralize SQL test dependencies version handling (elastic#40551) Mute testTracerLog Mute testHttpInput Include functions' aliases in the list of functions (elastic#40584) Optimise rejection of out-of-range `long` values (elastic#40325) Add docs for cluster.remote.*.proxy setting (elastic#40281) Migrate systemd packaging tests from bats to java (elastic#39954) Move top-level pipeline aggs out of QuerySearchResult (elastic#40319) Use openjdk 12 in packer cache script (elastic#40498) ...

jkakavas · 2019-03-29T06:18:47Z

@atorok, can we backport this to 7.0 and 6.7 too ? We'll still run these for some time in CI and parts of the changes here also affect/resolve issues with other test fixtures (idp-fixture)

Replaces the vagrant based kerberos fixtures with docker based test fixtures plugin. The configuration is now entirely static on the docker side and no longer driven by Gradle, also two different services are being configured since there are two different consumers of the fixture that can run in parallel and require different configurations.

* elastic/7.0: (50 commits) Fix more broken links in plugins docs Fix archives links in plugins docs Disable integTest when Docker is not available (elastic#40585) Add docs for bundled jdk (elastic#40487) [DOCS] Correct keystore commands for Email and Jira actions in Watcher (elastic#40417) (elastic#40613) Add usage indicators for the bundled JDK (elastic#40616) Add ability to mute and mute flaky fixture (elastic#40630) Test fixtures krb5 (elastic#40297) Update docs for the DFR similarity (elastic#40579) Update ingest jdocs that a null return value will drop the current document. (elastic#40359) [DOCS] Document common settings for snapshot repository plugins (elastic#40475) (elastic#40607) [DOCS] Fixes formatting in breaking changes Handle null retention leases in WaitForNoFollowersStep (elastic#40477) Correct ILM metadata minimum compatibility version (elastic#40569) Mute SpecificMasterNodesIT.testElectOnlyBetweenMasterNodes() Mute testHttpInput Include functions' aliases in the list of functions (elastic#40584) Optimise rejection of out-of-range `long` values (elastic#40325) Add docs for cluster.remote.*.proxy setting (elastic#40281) Mute WatchAckTests.testAckAllActions ...

alpar-t added 20 commits March 13, 2019 19:06

Upgrae plugin to latest and expose udp

e47fc1c

Proper dependencies preprocess task

a66b53c

Move fixture to docker

fecdd7a

Add two separate fixtures for the different configs used by this fixture.

Gradle cahnges without config errors

64b0c53

Genenrate conf files for fixture

7dfd48a

Fix typo

77b64e3

use the fixture

9865787

Use the pre process task

be20b9d

no more need to be explicit about evaluation

ab7dc63

Move hdfs to docker

a53134e

remove the vagrant file

463e5d2

disable fixture unit test

83536bc

Make sure post process runs before tests

98cd4ac

Remove unnecesary config

d0e716d

We only ever connect on localhost. This entry just causes a missleading connection error

Fix replacing the UDP port

27b05cd

inject ports before staring hdfs

ec3a166

Fix hdfs-repository tests

124918c

Fix kerberos tests

fcab3b0

turns out we need both local and mapped port in the file

Disable rest integ tests tasks for test fixtures

ffafbc1

We need to disable these too when docker is not supproted.

alpar-t added >non-issue :Delivery/Build Build or test infrastructure v8.0.0 v7.2.0 labels Mar 21, 2019

alpar-t requested review from jbaiera and mark-vieira March 21, 2019 09:05

mark-vieira requested changes Mar 21, 2019

View reviewed changes

Merge remote-tracking branch 'origin/master' into test-fixtures-krb5

8ae3de5

alpar-t added 2 commits March 25, 2019 15:09

Merge remote-tracking branch 'origin/master' into test-fixtures-krb5

50673c7

Merge remote-tracking branch 'origin/master' into test-fixtures-krb5

9a22dce

mark-vieira approved these changes Mar 25, 2019

View reviewed changes

jbaiera approved these changes Mar 25, 2019

View reviewed changes

alpar-t added 2 commits March 27, 2019 08:16

Fix failing test

330ff07

Merge remote-tracking branch 'origin/master' into test-fixtures-krb5

7866be8

This was referenced Mar 28, 2019

Add OIDC authentication Integration Tests #40262

Merged

Disable integTest when Docker is not available #40585

Merged

Merge remote-tracking branch 'origin/master' into test-fixtures-krb5

a65b0df

alpar-t merged commit 65e0317 into elastic:master Mar 28, 2019

alpar-t deleted the test-fixtures-krb5 branch March 28, 2019 15:23

alpar-t added v7.0.0 v6.7.1 labels Mar 29, 2019

jakelandis added v7.0.0-rc2 and removed v7.0.0 labels Apr 3, 2019

mark-vieira added the Team:Delivery Meta label for Delivery team label Nov 11, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test fixtures krb5 #40297

Test fixtures krb5 #40297

alpar-t commented Mar 21, 2019

elasticmachine commented Mar 21, 2019

alpar-t commented Mar 21, 2019

mark-vieira Mar 21, 2019

alpar-t Mar 22, 2019

mark-vieira Mar 22, 2019

mark-vieira Mar 21, 2019

alpar-t Mar 22, 2019

mark-vieira left a comment

alpar-t commented Mar 25, 2019

jbaiera left a comment

jbaiera Mar 25, 2019

jbaiera Mar 25, 2019

alpar-t Mar 27, 2019

alpar-t commented Mar 26, 2019

alpar-t commented Mar 27, 2019

alpar-t commented Mar 27, 2019

alpar-t commented Mar 27, 2019

alpar-t commented Mar 28, 2019

jkakavas commented Mar 29, 2019

@@ @@ -19,6 +19,9 @@ @@
               set -e
+              krb5kdc
+              kadmind

Test fixtures krb5 #40297

Test fixtures krb5 #40297

Conversation

alpar-t commented Mar 21, 2019

elasticmachine commented Mar 21, 2019

alpar-t commented Mar 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mark-vieira left a comment

Choose a reason for hiding this comment

alpar-t commented Mar 25, 2019

jbaiera left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alpar-t commented Mar 26, 2019

alpar-t commented Mar 27, 2019

alpar-t commented Mar 27, 2019

alpar-t commented Mar 27, 2019

alpar-t commented Mar 28, 2019

jkakavas commented Mar 29, 2019