Enhance `E2E` Test Suite to support Parallel Execution #833

seshachalam-yv · 2024-07-17T15:23:20Z

How to categorize this PR?

/area testing
/kind enhancement
What this PR does / why we need it:

This PR enhances the e2e test suite by:

Parallel Execution: Significantly reduces test execution time, decreasing overall duration from 25 minutes to 12 minutes.
Test Refactoring: Adopts new etcd naming conventions, improving test isolation and facilitating parallel execution.
Resource Initialization: Utilizes sync.Once for the safe and efficient initialization of shared resources, such as TLS secrets and bucket credentials, within the same namespace.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Why Use Unique Names for etcd Test Cases?

We opted for unique names instead of unique namespaces for each etcd test case to streamline DNS management. If each test used a unique namespace, it would require numerous DNS entries in server.csr.conf, such as etcd-{provider}.{NAMESPACE}. Using a constant namespace simplifies DNS configuration and resource management across different providers.

Release note:

Enhanced parallel execution support in e2e tests, reducing time and improving test suite robustness.

gardener-prow · 2024-07-17T15:23:23Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

gitguardian · 2024-07-17T15:23:56Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
-	-	Generic Private Key	`ef0c014`	test/e2e/resources/tls/server.key	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

ashwani2k · 2024-08-06T09:56:51Z

Tested this change and it runs now in 4 processes for the tests and completed in ~10mins.

-------------------
Running e2e tests
-------------------
Running Suite: e2e Suite - ~/go/src/github.com/gardener/etcd-druid/test/e2e
========================================================================================
Random Seed: 1722934272 - will randomize all specs

Will run 8 of 8 specs
Running in parallel across 4 processes
........ 
... 
.........
------------------------------

Ran 8 of 8 Specs in 605.391 seconds
SUCCESS! -- 8 Passed | 0 Failed | 0 Pending | 0 Skipped


Ginkgo ran 1 suite in 10m11.882335208s
Test Suite Passed

Running the same test on the master branch finished in 23mins.

Ran 7 of 7 Specs in 1425.432 seconds
SUCCESS! -- 7 Passed | 0 Failed | 0 Pending | 0 Skipped
--- PASS: TestIntegration (1425.43s)
PASS
ok      github.com/gardener/etcd-druid/test/e2e 1426.051s

This is cool and time saver.

unmarshall

Thanks for the PR. In general i would recommend that ONLY refactorings which are absolutely essential for ensuring that the tests run concurrently should be kept.

hack/e2e-test/run-e2e-test.sh

test/e2e/etcd_backup_test.go

test/e2e/etcd_compaction_test.go

test/e2e/resources/tls/server.csr.conf

test/e2e/suite_test.go

Ensure that the GINKGO binary is required for running the ci-e2e-kind and ci-e2e-kind-azure targets.

…est case

- Introduce sync.Once to ensure secrets are deployed exactly once across concurrent tests, enhancing efficiency and avoiding redundant operations. - Implement checks to update secrets if they already exist, rather than failing on re-deployment. - Revise the deletion strategy for namespaces to ensure they are removed only after all associated etcd resources are verified to be deleted. - Refactor the zero-downtime validation job to improve readability and maintainability, aligning with best practices for script execution in test scenarios.

- Used unique etcd instance names per test case to prevent collisions in shared namespace, improving test isolation. - Refactor cleanup processes to ensure etcd instances are deleted at the end of each test.

unmarshall · 2024-08-15T08:12:02Z

I checked out this PR and ran make ci-e2e-kind - i ran twice but there is one single test that consistently fails.

[FAILED] in [It] - /Users/i062009/go/src/github.com/gardener/etcd-repos/etcd-druid/test/e2e/etcd_compaction_test.go:156 @ 08/15/24 13:14:26.564
  << Timeline

  [FAILED] Timed out after 180.001s.
  Expected
      <*errors.errorString | 0x14000a6e140>:
      latest delta snapshot count is not 0
      {
          s: "latest delta snapshot count is not 0",
      }
  to be nil
  In [It] at: /Users/i062009/go/src/github.com/gardener/etcd-repos/etcd-druid/test/e2e/etcd_compaction_test.go:156 @ 08/15/24 13:14:26.564

I know this PR does not resolve flaky e2e tests but only improves its run time. Just reporting in this ticket.

seshachalam-yv · 2024-08-16T06:58:36Z

I ran the tests locally and they passed.

Ran 8 of 8 Specs in 494.086 seconds
SUCCESS! -- 8 Passed | 0 Failed | 0 Pending | 0 Skipped

-------------------
Running e2e tests
-------------------
Running Suite: e2e Suite - /Users/i568019/Desktop/review/etcd-druid/test/e2e
============================================================================
Random Seed: 1723790609

Will run 8 of 8 specs
Running in parallel across 9 processes
------------------------------
[BeforeSuite] PASSED [0.042 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.044 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.043 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.044 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.044 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.044 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.043 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.044 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[BeforeSuite] PASSED [0.061 seconds]
[BeforeSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:65

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating namespace","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"ca-etcd-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-server-cert-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-client-tls-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"creating secret","secret":{"name":"etcd-backup-aws","namespace":"shoot"}}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
• [186.702 seconds]
Etcd Backup when single-node etcd is configured with provider aws Should create, test backup and delete etcd with backup
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_backup_test.go:58

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.094
  STEP: Create etcd @ 08/16/24 12:13:37.153
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"etcd is ready","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"Checking statefulset","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"Checking configmap","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"Checking client service","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  STEP: Create debug pod @ 08/16/24 12:15:31.936
  STEP: Check initial snapshot is available @ 08/16/24 12:15:37.969
  STEP: Put keys into etcd @ 08/16/24 12:15:38.036
  {"level":"info","ts":"2024-08-16T12:15:38+05:30","msg":"populating etcd with sequential key-value pairs","fromKey":"foo-1","fromValue":"bar-1","toKey":"foo-10","toValue":"bar-10"}
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"put key-value successful","key":"foo-1","value":"bar-1"}
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"put key-value successful","key":"foo-2","value":"bar-2"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-3","value":"bar-3"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-4","value":"bar-4"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-5","value":"bar-5"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-6","value":"bar-6"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-7","value":"bar-7"}
  {"level":"info","ts":"2024-08-16T12:15:42+05:30","msg":"put key-value successful","key":"foo-8","value":"bar-8"}
  {"level":"info","ts":"2024-08-16T12:15:42+05:30","msg":"put key-value successful","key":"foo-9","value":"bar-9"}
  {"level":"info","ts":"2024-08-16T12:15:42+05:30","msg":"put key-value successful","key":"foo-10","value":"bar-10"}
  STEP: Check snapshot after putting data into etcd @ 08/16/24 12:15:42.592
  STEP: Trigger on-demand full snapshot @ 08/16/24 12:15:42.694
  STEP: Put additional data into etcd @ 08/16/24 12:15:42.777
  {"level":"info","ts":"2024-08-16T12:15:42+05:30","msg":"populating etcd with sequential key-value pairs","fromKey":"foo-11","fromValue":"bar-11","toKey":"foo-15","toValue":"bar-15"}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"put key-value successful","key":"foo-11","value":"bar-11"}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"put key-value successful","key":"foo-12","value":"bar-12"}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"put key-value successful","key":"foo-13","value":"bar-13"}
  {"level":"info","ts":"2024-08-16T12:15:46+05:30","msg":"put key-value successful","key":"foo-14","value":"bar-14"}
  {"level":"info","ts":"2024-08-16T12:15:46+05:30","msg":"put key-value successful","key":"foo-15","value":"bar-15"}
  STEP: Trigger on-demand delta snapshot @ 08/16/24 12:15:46.36
  STEP: Test cluster restoration by deleting data directory @ 08/16/24 12:15:46.41
  {"level":"info","ts":"2024-08-16T12:15:46+05:30","msg":"waiting for sts to become unready","statefulSetName":"etcd-single-node-aws"}
  {"level":"info","ts":"2024-08-16T12:16:18+05:30","msg":"sts is unready","statefulSetName":"etcd-single-node-aws"}
  {"level":"info","ts":"2024-08-16T12:16:18+05:30","msg":"waiting for sts to become ready again","statefulSetName":"etcd-single-node-aws"}
  {"level":"info","ts":"2024-08-16T12:16:36+05:30","msg":"sts is ready","statefulSetName":"etcd-single-node-aws"}
  {"level":"info","ts":"2024-08-16T12:16:36+05:30","msg":"fetching etcd key-value pairs"}
  {"level":"info","ts":"2024-08-16T12:16:36+05:30","msg":"fetched key-value pair from etcd","key":"foo-1","value":"bar-1"}
  {"level":"info","ts":"2024-08-16T12:16:37+05:30","msg":"fetched key-value pair from etcd","key":"foo-2","value":"bar-2"}
  {"level":"info","ts":"2024-08-16T12:16:37+05:30","msg":"fetched key-value pair from etcd","key":"foo-3","value":"bar-3"}
  {"level":"info","ts":"2024-08-16T12:16:37+05:30","msg":"fetched key-value pair from etcd","key":"foo-4","value":"bar-4"}
  {"level":"info","ts":"2024-08-16T12:16:37+05:30","msg":"fetched key-value pair from etcd","key":"foo-5","value":"bar-5"}
  {"level":"info","ts":"2024-08-16T12:16:37+05:30","msg":"fetched key-value pair from etcd","key":"foo-6","value":"bar-6"}
  {"level":"info","ts":"2024-08-16T12:16:38+05:30","msg":"fetched key-value pair from etcd","key":"foo-7","value":"bar-7"}
  {"level":"info","ts":"2024-08-16T12:16:38+05:30","msg":"fetched key-value pair from etcd","key":"foo-8","value":"bar-8"}
  {"level":"info","ts":"2024-08-16T12:16:38+05:30","msg":"fetched key-value pair from etcd","key":"foo-9","value":"bar-9"}
  {"level":"info","ts":"2024-08-16T12:16:38+05:30","msg":"fetched key-value pair from etcd","key":"foo-10","value":"bar-10"}
  {"level":"info","ts":"2024-08-16T12:16:38+05:30","msg":"fetched key-value pair from etcd","key":"foo-11","value":"bar-11"}
  {"level":"info","ts":"2024-08-16T12:16:39+05:30","msg":"fetched key-value pair from etcd","key":"foo-12","value":"bar-12"}
  {"level":"info","ts":"2024-08-16T12:16:39+05:30","msg":"fetched key-value pair from etcd","key":"foo-13","value":"bar-13"}
  {"level":"info","ts":"2024-08-16T12:16:39+05:30","msg":"fetched key-value pair from etcd","key":"foo-14","value":"bar-14"}
  {"level":"info","ts":"2024-08-16T12:16:39+05:30","msg":"fetched key-value pair from etcd","key":"foo-15","value":"bar-15"}
  STEP: Deleting debug pod @ 08/16/24 12:16:39.766
  STEP: Deleting etcd @ 08/16/24 12:16:39.769
  {"level":"info","ts":"2024-08-16T12:16:39+05:30","msg":"Checking if etcd is gone","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:43+05:30","msg":"Checking if statefulset is gone","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:43+05:30","msg":"Checking if configmap is gone","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:43+05:30","msg":"Checking client service is gone","etcd":{"name":"etcd-single-node-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:43+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
• [199.352 seconds]
Etcd Single-node etcd configuration Scaling up from single-node to multi-node without TLS should scale a single-node etcd (TLS not enabled for peerUrl) to a multi-node etcd cluster (TLS not enabled for peerUrl)
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_multi_node_test.go:139

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.096
  STEP: Creating a single-node etcd @ 08/16/24 12:13:37.152
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  STEP: Scaling up a healthy cluster (from 1 to 3 replicas) @ 08/16/24 12:15:25.895
  {"level":"info","ts":"2024-08-16T12:16:32+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:52+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:52+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:52+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:52+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  STEP: Deleting etcd @ 08/16/24 12:16:52.398
  {"level":"info","ts":"2024-08-16T12:16:52+05:30","msg":"Checking if etcd is gone","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:56+05:30","msg":"Checking if statefulset is gone","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:56+05:30","msg":"Checking if configmap is gone","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:56+05:30","msg":"Checking client service is gone","etcd-multi-node":{"name":"scale-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:56+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
• [235.815 seconds]
Etcd Compaction when compaction is enabled for single-node etcd with provider aws should test compaction on backup
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_compaction_test.go:52

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.097
  STEP: Create etcd @ 08/16/24 12:13:37.153
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"etcd is ready","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"Checking statefulset","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"Checking configmap","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:25+05:30","msg":"Checking client service","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  STEP: Create debug pod @ 08/16/24 12:15:25.895
  STEP: Check initial snapshot is available @ 08/16/24 12:15:35.935
  STEP: Put keys into etcd @ 08/16/24 12:15:36.007
  {"level":"info","ts":"2024-08-16T12:15:36+05:30","msg":"populating etcd with sequential key-value pairs","fromKey":"foo-1","fromValue":"bar-1","toKey":"foo-10","toValue":"bar-10"}
  {"level":"info","ts":"2024-08-16T12:15:39+05:30","msg":"put key-value successful","key":"foo-1","value":"bar-1"}
  {"level":"info","ts":"2024-08-16T12:15:39+05:30","msg":"put key-value successful","key":"foo-2","value":"bar-2"}
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"put key-value successful","key":"foo-3","value":"bar-3"}
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"put key-value successful","key":"foo-4","value":"bar-4"}
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"put key-value successful","key":"foo-5","value":"bar-5"}
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"put key-value successful","key":"foo-6","value":"bar-6"}
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"put key-value successful","key":"foo-7","value":"bar-7"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-8","value":"bar-8"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-9","value":"bar-9"}
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"put key-value successful","key":"foo-10","value":"bar-10"}
  STEP: Check snapshot after putting data into etcd @ 08/16/24 12:15:41.477
  STEP: Put additional data into etcd @ 08/16/24 12:15:41.585
  {"level":"info","ts":"2024-08-16T12:15:41+05:30","msg":"populating etcd with sequential key-value pairs","fromKey":"foo-11","fromValue":"bar-11","toKey":"foo-15","toValue":"bar-15"}
  {"level":"info","ts":"2024-08-16T12:15:44+05:30","msg":"put key-value successful","key":"foo-11","value":"bar-11"}
  {"level":"info","ts":"2024-08-16T12:15:44+05:30","msg":"put key-value successful","key":"foo-12","value":"bar-12"}
  {"level":"info","ts":"2024-08-16T12:15:44+05:30","msg":"put key-value successful","key":"foo-13","value":"bar-13"}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"put key-value successful","key":"foo-14","value":"bar-14"}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"put key-value successful","key":"foo-15","value":"bar-15"}
  STEP: Trigger on-demand delta snapshot @ 08/16/24 12:15:45.281
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"waiting for compaction job to become successful"}
  {"level":"info","ts":"2024-08-16T12:16:18+05:30","msg":"compaction job is successful"}
  STEP: Put additional data into etcd @ 08/16/24 12:16:18.649
  {"level":"info","ts":"2024-08-16T12:16:18+05:30","msg":"populating etcd with sequential key-value pairs","fromKey":"foo-16","fromValue":"bar-16","toKey":"foo-20","toValue":"bar-20"}
  {"level":"info","ts":"2024-08-16T12:17:23+05:30","msg":"put key-value successful","key":"foo-16","value":"bar-16"}
  {"level":"info","ts":"2024-08-16T12:17:24+05:30","msg":"put key-value successful","key":"foo-17","value":"bar-17"}
  {"level":"info","ts":"2024-08-16T12:17:24+05:30","msg":"put key-value successful","key":"foo-18","value":"bar-18"}
  {"level":"info","ts":"2024-08-16T12:17:24+05:30","msg":"put key-value successful","key":"foo-19","value":"bar-19"}
  {"level":"info","ts":"2024-08-16T12:17:24+05:30","msg":"put key-value successful","key":"foo-20","value":"bar-20"}
  STEP: Trigger next on-demand delta snapshot @ 08/16/24 12:17:24.761
  STEP: Verify that there are new delta snapshots as compaction is not triggered yet because delta events have not reached next 15 revision @ 08/16/24 12:17:24.813
  STEP: Deleting debug pod @ 08/16/24 12:17:24.867
  STEP: Deleting etcd @ 08/16/24 12:17:24.87
  {"level":"info","ts":"2024-08-16T12:17:24+05:30","msg":"Checking if etcd is gone","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:32+05:30","msg":"Checking if statefulset is gone","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:32+05:30","msg":"Checking if configmap is gone","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:32+05:30","msg":"Checking client service is gone","etcd":{"name":"etcd-compaction-aws","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:32+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
• [237.517 seconds]
Etcd Single-node etcd configuration Scaling up from single-node to multi-node with TLS executes scaling up with TLS
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_multi_node_test.go:165

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.096
  STEP: Creating a single-node etcd @ 08/16/24 12:13:37.152
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:43+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:43+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:43+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:43+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  STEP: Scaling up a healthy cluster (from 1 to 3 replicas) with TLS enabled for peerUrl @ 08/16/24 12:15:43.989
  {"level":"info","ts":"2024-08-16T12:15:46+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:30+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:30+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:30+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:30+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  STEP: Deleting etcd @ 08/16/24 12:17:30.562
  {"level":"info","ts":"2024-08-16T12:17:30+05:30","msg":"Checking if etcd is gone","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:34+05:30","msg":"Checking if statefulset is gone","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:34+05:30","msg":"Checking if configmap is gone","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:34+05:30","msg":"Checking client service is gone","etcd-multi-node":{"name":"scale-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:34+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
• [245.561 seconds]
Etcd Single-node etcd configuration Scaling down from single-node to zero and back up without TLS executes scaling down to 0 and back up to 3 replicas without TLS
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_multi_node_test.go:190

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.096
  STEP: Creating a single-node etcd @ 08/16/24 12:13:37.152
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:45+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  STEP: Scaling down a healthy cluster (from 1 to 0 replica) @ 08/16/24 12:15:45.994
  {"level":"info","ts":"2024-08-16T12:15:48+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:52+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:52+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:52+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:52+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  STEP: Scaling up cluster (from 0 to 1 replica) @ 08/16/24 12:15:52.061
  {"level":"info","ts":"2024-08-16T12:15:54+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:16+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:16+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:16+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:16+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  STEP: Scaling up a healthy cluster (from 1 to 3 replica) @ 08/16/24 12:16:16.203
  {"level":"info","ts":"2024-08-16T12:17:16+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:36+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:36+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:36+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:36+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  STEP: Deleting etcd @ 08/16/24 12:17:36.584
  {"level":"info","ts":"2024-08-16T12:17:36+05:30","msg":"Checking if etcd is gone","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:42+05:30","msg":"Checking if statefulset is gone","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:42+05:30","msg":"Checking if configmap is gone","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:42+05:30","msg":"Checking client service is gone","etcd-multi-node":{"name":"scale-down-and-up-non-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:42+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
• [255.586 seconds]
Etcd Single-node etcd configuration Scaling down from single-node to zero and back up with TLS should scale down a single-node etcd to 0 replica, then scale up from 0->1 replica and then from 1->3 replicas with TLS enabled for cluster peerUrl
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_multi_node_test.go:225

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.097
  STEP: Creating a single-node etcd @ 08/16/24 12:13:37.153
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:31+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  STEP: Scaling down a healthy cluster (from 1 to 0 replica) @ 08/16/24 12:15:31.933
  {"level":"info","ts":"2024-08-16T12:15:33+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:37+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:37+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:37+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:15:37+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  STEP: Scaling up cluster (from 0 to 1 replica) @ 08/16/24 12:15:37.991
  {"level":"info","ts":"2024-08-16T12:15:40+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:02+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:02+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:02+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:16:02+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  STEP: Scaling up a healthy cluster (from 1 to 3 replica) with TLS enabled for peerUrl @ 08/16/24 12:16:02.101
  {"level":"info","ts":"2024-08-16T12:16:04+05:30","msg":"Waiting for etcd to become ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:48+05:30","msg":"etcd is ready","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:48+05:30","msg":"Checking statefulset","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:48+05:30","msg":"Checking configmap","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:48+05:30","msg":"Checking client service","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  STEP: Deleting etcd @ 08/16/24 12:17:48.635
  {"level":"info","ts":"2024-08-16T12:17:48+05:30","msg":"Checking if etcd is gone","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:52+05:30","msg":"Checking if statefulset is gone","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:52+05:30","msg":"Checking if configmap is gone","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:52+05:30","msg":"Checking client service is gone","etcd-multi-node":{"name":"scale-down-and-up-tls","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:52+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
• [418.764 seconds]
Etcd when multi-node is configured should perform etcd operations excluding zero downtime maintenance
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_multi_node_test.go:52

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.096
  STEP: Creating etcd @ 08/16/24 12:13:37.151
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:20+05:30","msg":"etcd is ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:20+05:30","msg":"Checking statefulset","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:20+05:30","msg":"Checking configmap","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:20+05:30","msg":"Checking client service","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  STEP: Hibernating etcd (Scaling down from 3 replicas to 0) @ 08/16/24 12:17:20.513
  {"level":"info","ts":"2024-08-16T12:17:20+05:30","msg":"Waiting for statefulset spec to reflect change in replicas to 0","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:22+05:30","msg":"Checking etcd","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:24+05:30","msg":"Checking statefulset","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:24+05:30","msg":"etcd is hibernated","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  STEP: Waking up etcd (Scaling up from 0 to 3 replicas) @ 08/16/24 12:17:24.547
  {"level":"info","ts":"2024-08-16T12:17:26+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:55+05:30","msg":"etcd is ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:55+05:30","msg":"Checking statefulset","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:55+05:30","msg":"Checking configmap","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:55+05:30","msg":"Checking client service","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  STEP: Member restart with data-dir/pvc intact @ 08/16/24 12:18:55.024
  {"level":"info","ts":"2024-08-16T12:18:55+05:30","msg":"Deleting one member pod","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:55+05:30","msg":"waiting for sts to become unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:57+05:30","msg":"sts is unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:57+05:30","msg":"Waiting for at least one etcd member to become unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:01+05:30","msg":"at least one etcd member is unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:01+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:21+05:30","msg":"etcd is ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:21+05:30","msg":"Checking statefulset","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:21+05:30","msg":"Checking configmap","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:21+05:30","msg":"Checking client service","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  STEP: Single member restoration @ 08/16/24 12:19:21.256
  {"level":"info","ts":"2024-08-16T12:19:21+05:30","msg":"Creating debug pod","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"Deleting member directory of one member pod","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"waiting for sts to become unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:37+05:30","msg":"sts is unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:37+05:30","msg":"Waiting for at least one etcd member to become unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:41+05:30","msg":"at least one etcd member is unready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:41+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:31+05:30","msg":"etcd is ready","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:31+05:30","msg":"Checking statefulset","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:31+05:30","msg":"Checking configmap","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:31+05:30","msg":"Checking client service","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  STEP: Deleting debug pod @ 08/16/24 12:20:31.803
  STEP: Deleting etcd @ 08/16/24 12:20:31.809
  {"level":"info","ts":"2024-08-16T12:20:31+05:30","msg":"Checking if etcd is gone","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:35+05:30","msg":"Checking if statefulset is gone","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:35+05:30","msg":"Checking if configmap is gone","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:35+05:30","msg":"Checking client service is gone","etcd":{"name":"multi-node-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:20:35+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
• [449.707 seconds]
Etcd when multi-node is configured should perform zero downtime maintenance operation: defragmentation
/Users/i568019/Desktop/review/etcd-druid/test/e2e/etcd_multi_node_test.go:95

  Timeline >>
  STEP: Purge snapstore @ 08/16/24 12:13:37.096
  STEP: Creating etcd @ 08/16/24 12:13:37.152
  {"level":"info","ts":"2024-08-16T12:13:37+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:04+05:30","msg":"etcd is ready","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:04+05:30","msg":"Checking statefulset","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:04+05:30","msg":"Checking configmap","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:17:04+05:30","msg":"Checking client service","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  STEP: Deploying etcd zero downtime validator job @ 08/16/24 12:17:04.454
  {"level":"info","ts":"2024-08-16T12:17:04+05:30","msg":"Creating job to ensure etcd zero downtime","job":"etcd-zero-down-time-validator-rolling-update"}
  {"level":"info","ts":"2024-08-16T12:17:16+05:30","msg":"Job is ready","job":"etcd-zero-down-time-validator-rolling-update"}
  STEP: Conducting zero downtime rolling updates @ 08/16/24 12:17:16.505
  {"level":"info","ts":"2024-08-16T12:17:54+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:20+05:30","msg":"etcd is ready","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:20+05:30","msg":"Checking statefulset","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:20+05:30","msg":"Checking configmap","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:20+05:30","msg":"Checking client service","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:18:20+05:30","msg":"Etcd Cluster is healthy and there is no downtime","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  STEP: Performing zero downtime maintenance operation: defragmentation @ 08/16/24 12:18:20.859
  {"level":"info","ts":"2024-08-16T12:18:20+05:30","msg":"Configuring defragmentation schedule for every 1 minute","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:05+05:30","msg":"Waiting for etcd to become ready","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"etcd is ready","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"Checking statefulset","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"Checking configmap","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"Checking client service","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"Etcd Cluster is healthy and there is no downtime","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:19:25+05:30","msg":"Waiting for defragmentation to finish","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:21:00+05:30","msg":"Defragmentation is finished","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:21:00+05:30","msg":"Checking for any Etcd downtime","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:21:00+05:30","msg":"Etcd Cluster is healthy and there is no downtime","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  STEP: Deleting etcd @ 08/16/24 12:21:00.729
  {"level":"info","ts":"2024-08-16T12:21:00+05:30","msg":"Checking if etcd is gone","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:21:06+05:30","msg":"Checking if statefulset is gone","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:21:06+05:30","msg":"Checking if configmap is gone","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:21:06+05:30","msg":"Checking client service is gone","etcd":{"name":"defrag-zero-downtime-etcd","namespace":"shoot"}}
  {"level":"info","ts":"2024-08-16T12:21:06+05:30","msg":"Deleting PVCs"}
  << Timeline
------------------------------
[SynchronizedAfterSuite] PASSED [0.000 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115
------------------------------
[SynchronizedAfterSuite] PASSED [294.685 seconds]
[SynchronizedAfterSuite] 
/Users/i568019/Desktop/review/etcd-druid/test/e2e/suite_test.go:115

  Timeline >>
  {"level":"info","ts":"2024-08-16T12:21:06+05:30","msg":"Checking for Etcd resources before deleting namespace","namespace":"shoot","namespace":"shoot"}
  {"level":"info","ts":"2024-08-16T12:21:06+05:30","msg":"No Etcd resources found; proceeding with namespace deletion","namespace":"shoot","namespace":"shoot"}
  << Timeline
------------------------------

Ran 8 of 8 Specs in 494.086 seconds
SUCCESS! -- 8 Passed | 0 Failed | 0 Pending | 0 Skipped


Ginkgo ran 1 suite in 8m21.349894625s
Test Suite Passed
Exporting logs for cluster "etcd-druid-e2e" to:
/tmp/etcd-druid-e2e
/Users/i568019/Desktop/review/etcd-druid/hack/tools/bin/kind delete cluster --name etcd-druid-e2e
Deleting cluster "etcd-druid-e2e" ...
Deleted nodes: ["etcd-druid-e2e-control-plane"]

gardener-robot added area/testing Testing related kind/enhancement Enhancement, improvement, extension needs/review Needs review size/xl Size of pull request is huge (see gardener-robot robot/bots/size.py) needs/second-opinion Needs second review by someone else labels Jul 17, 2024

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 17, 2024

gardener-robot-ci-2 added needs/ok-to-test Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 17, 2024

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 17, 2024

seshachalam-yv force-pushed the refactor/test-parallel-execution branch from 79c33f7 to 0946ea4 Compare July 17, 2024 16:00

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 17, 2024

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 17, 2024

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 17, 2024

seshachalam-yv force-pushed the refactor/test-parallel-execution branch from 4234b88 to c0837d3 Compare July 29, 2024 11:03

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 29, 2024

seshachalam-yv force-pushed the refactor/test-parallel-execution branch from c0837d3 to 13d8853 Compare July 29, 2024 15:00

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 29, 2024

gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jul 29, 2024

seshachalam-yv marked this pull request as ready for review July 29, 2024 15:18

seshachalam-yv requested a review from a team as a code owner July 29, 2024 15:18

seshachalam-yv changed the title ~~Refactor e2e tests for improved parallel execution support~~ Enhance E2E Test Suite to support Parallel Execution Jul 29, 2024

gardener deleted a comment from gardener-prow bot Jul 29, 2024

gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Jul 29, 2024

gardener deleted a comment from gardener-prow bot Jul 29, 2024

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 5, 2024

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 5, 2024

ishan16696 assigned unmarshall and unassigned shreyas-s-rao Aug 8, 2024

unmarshall requested changes Aug 9, 2024

View reviewed changes

gardener-robot added the needs/changes Needs (more) changes label Aug 9, 2024

gardener-robot-ci-3 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 9, 2024

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 9, 2024

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 9, 2024

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Aug 9, 2024

seshachalam-yv added 7 commits August 12, 2024 12:40

Add GINKGO dependency to ci-e2e-kind targets in Makefile

335940f

Ensure that the GINKGO binary is required for running the ci-e2e-kind and ci-e2e-kind-azure targets.

Update e2e test script to use ginkgo with enhanced options

c9d078b

Update server.csr.conf with comprehensive DNS entries for each etcd t…

ef0c014

…est case

Enhance etcd e2e tests for improved isolation and parallel execution

bd4b9a7

- Used unique etcd instance names per test case to prevent collisions in shared namespace, improving test isolation. - Refactor cleanup processes to ensure etcd instances are deleted at the end of each test.

Reduce Ginkgo parallel processes from 5 to 4 for e2e tests

0a1edad

Incorporate feedback from Madhav's review

801abdf

seshachalam-yv force-pushed the refactor/test-parallel-execution branch from 7509f93 to 801abdf Compare August 12, 2024 07:13

gardener-robot-ci-1 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels Aug 12, 2024

gardener deleted a comment from gardener-prow bot Aug 12, 2024

seshachalam-yv requested a review from unmarshall August 12, 2024 08:12

unmarshall mentioned this pull request Aug 13, 2024

☂️ Enhance and Stabilise Druid E2E tests #782

Open

18 tasks

gardener deleted a comment from gardener-robot Aug 16, 2024

unmarshall approved these changes Aug 16, 2024

View reviewed changes

unmarshall merged commit 3383e02 into gardener:master Aug 16, 2024
11 checks passed

gardener-robot added the status/closed Issue is closed (either delivered or triaged) label Aug 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance `E2E` Test Suite to support Parallel Execution #833

Enhance `E2E` Test Suite to support Parallel Execution #833

seshachalam-yv commented Jul 17, 2024 •

edited

Loading

gardener-prow bot commented Jul 17, 2024

gitguardian bot commented Jul 17, 2024 •

edited

Loading

ashwani2k commented Aug 6, 2024

unmarshall left a comment

unmarshall commented Aug 15, 2024 •

edited

Loading

seshachalam-yv commented Aug 16, 2024 •

edited

Loading

Enhance E2E Test Suite to support Parallel Execution #833

Enhance E2E Test Suite to support Parallel Execution #833

Conversation

seshachalam-yv commented Jul 17, 2024 • edited Loading

gardener-prow bot commented Jul 17, 2024

gitguardian bot commented Jul 17, 2024 • edited Loading

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

ashwani2k commented Aug 6, 2024

unmarshall left a comment

Choose a reason for hiding this comment

unmarshall commented Aug 15, 2024 • edited Loading

seshachalam-yv commented Aug 16, 2024 • edited Loading

Enhance `E2E` Test Suite to support Parallel Execution #833

Enhance `E2E` Test Suite to support Parallel Execution #833

seshachalam-yv commented Jul 17, 2024 •

edited

Loading

gitguardian bot commented Jul 17, 2024 •

edited

Loading

unmarshall commented Aug 15, 2024 •

edited

Loading

seshachalam-yv commented Aug 16, 2024 •

edited

Loading