Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use healthcheck cron job to determine cluster readiness #743

Merged
merged 16 commits into from
Mar 30, 2021
Merged

Use healthcheck cron job to determine cluster readiness #743

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
26f7164
feat: provide a healthcheck that replies on cron job
whereswaldon Mar 10, 2021
fd7fb75
wip: attempt to watch healthcheck job to determine cluster readiness
whereswaldon Mar 10, 2021
f8f10e6
feat: fix watch usage and pass proper client
whereswaldon Mar 11, 2021
420e4f1
feat: log when health check passes
whereswaldon Mar 12, 2021
cd7b67e
fix: ensure we cancel context to avoid leak
whereswaldon Mar 12, 2021
c6dc076
feat: warn that skip health checks config is a no-op
whereswaldon Mar 15, 2021
2c1a44c
fix: improve docs and log output as per code review
whereswaldon Mar 16, 2021
1325f9e
feat: update job monitoring logic to match new expected behavior
whereswaldon Mar 22, 2021
0d93eec
feat: add config knob for healthcheck timeout
whereswaldon Mar 29, 2021
8fef0ba
feat: use healthcheck timeout knob in e2e suite
whereswaldon Mar 29, 2021
359eae4
fix: only print usage warning when needed
whereswaldon Mar 29, 2021
dd9dac3
Merge branch 'main' into SDCICD-577
whereswaldon Mar 30, 2021
6de4970
feat: fail beforesuite if duration parse fails
whereswaldon Mar 30, 2021
7dda5b8
feat: repurpose old skip healthcheck flag to skip new ones
whereswaldon Mar 30, 2021
71de624
feat: log when skipping health checks
whereswaldon Mar 30, 2021
e0e0143
fix: actually fail healthcheck if any part fails
whereswaldon Mar 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 20 additions & 10 deletions pkg/common/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,12 @@ var Tests = struct {
// Env: SKIP_CLUSTER_HEALTH_CHECKS
SkipClusterHealthChecks string

// ClusterHealthChecksTimeout defines the duration for which the harness will
// wait for the cluster to indicate it is healthy before cancelling the test
// run. This value should be formatted for use with time.ParseDuration.
// Env: CLUSTER_HEALTH_CHECKS_TIMEOUT
ClusterHealthChecksTimeout string

// MetricsBucket is the bucket that metrics data will be uploaded to.
// Env: METRICS_BUCKET
MetricsBucket string
Expand All @@ -184,16 +190,17 @@ var Tests = struct {
ServiceAccount string
}{

PollingTimeout: "tests.pollingTimeout",
GinkgoSkip: "tests.ginkgoSkip",
GinkgoFocus: "tests.focus",
TestsToRun: "tests.testsToRun",
SuppressSkipNotifications: "tests.suppressSkipNotifications",
CleanRuns: "tests.cleanRuns",
OperatorSkip: "tests.operatorSkip",
SkipClusterHealthChecks: "tests.skipClusterHealthChecks",
MetricsBucket: "tests.metricsBucket",
ServiceAccount: "tests.serviceAccount",
PollingTimeout: "tests.pollingTimeout",
GinkgoSkip: "tests.ginkgoSkip",
GinkgoFocus: "tests.focus",
TestsToRun: "tests.testsToRun",
SuppressSkipNotifications: "tests.suppressSkipNotifications",
CleanRuns: "tests.cleanRuns",
OperatorSkip: "tests.operatorSkip",
SkipClusterHealthChecks: "tests.skipClusterHealthChecks",
MetricsBucket: "tests.metricsBucket",
ServiceAccount: "tests.serviceAccount",
ClusterHealthChecksTimeout: "tests.clusterHealthChecksTimeout",
}

// Cluster config keys.
Expand Down Expand Up @@ -527,6 +534,9 @@ func init() {
viper.SetDefault(Tests.SkipClusterHealthChecks, false)
viper.BindEnv(Tests.OperatorSkip, "SKIP_CLUSTER_HEALTH_CHECKS")

viper.SetDefault(Tests.ClusterHealthChecksTimeout, "2h")
viper.BindEnv(Tests.ClusterHealthChecksTimeout, "CLUSTER_HEALTH_CHECKS_TIMEOUT")

viper.SetDefault(Tests.MetricsBucket, "osde2e-metrics")
viper.BindEnv(Tests.MetricsBucket, "METRICS_BUCKET")

Expand Down
11 changes: 8 additions & 3 deletions pkg/e2e/e2e.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,13 +132,18 @@ func beforeSuite() bool {
if err != nil {
log.Printf("Error generating Kube Clientset: %v\n", err)
}
ctx, cancel := context.WithTimeout(context.Background(), time.Hour*2)
duration, err := time.ParseDuration(viper.GetString(config.Tests.ClusterHealthChecksTimeout))
if err != nil {
log.Printf("Failed parsing health check timeout, using 2 hours: %v", err)
duration = time.Hour * 2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-defaulting this will mask certain configuration and coding errors. IMHO it would be better to just blow up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, It is done.

}
ctx, cancel := context.WithTimeout(context.Background(), duration)
defer cancel()
if viper.GetString(config.Tests.SkipClusterHealthChecks) != "" {
if viper.GetString(config.Tests.SkipClusterHealthChecks) != "false" {
log.Println("WARNING: Skipping cluster health checks is no longer supported, as they no longer introduce delay into the build. Ignoring your request to skip them.")
}
err = healthchecks.CheckHealthcheckJob(kubeClient, ctx, nil)
events.HandleErrorWithEvents(err, events.HealthCheckSuccessful, events.HealthCheckFailed).ShouldNot(HaveOccurred(), "cluster failed health check")
events.HandleErrorWithEvents(err, events.HealthCheckSuccessful, events.HealthCheckFailed)
if err != nil {
log.Printf("Cluster failed health check: %v", err)
getLogs()
Expand Down