E2E: Groundwork for PR runs

[dereknola]

Proposed Changes

This PR address several issues that need to be resolved before we add E2E tests to PR checks.

• Converted CreateCluster and CreateLocalCluster to provision nodes in parallel. This improves test time significantly, provisioning the VMs previously was taking 40-60% of the runtime of a test. Testing on my local machine, validatecluster_test.go went from 10m29s --> 5m28s. E2E Runners are much slower, but saw improvement from 40m -> 20m.
• Add local flag to validatecluster test. This allows the test to deploy a prebuild K3s binary, i.e. the binary associated with the PR. This test will be the first that gets converted to run on PRs. The remaining tests will be converted later.
• Converted all E2E tests to Ordered tests. This has already been changed for integration tests. This bring the benefit of assuming that all It clauses should be run sequentially, and that an error in one causes any future It clauses to be skipped. Previously, It clauses would still run after the failure and usually fail as well. Reading GH logs on a failure is much easier.
• Fixes and improvements to the init.sh and run_test.sh scripts which are deployed on E2E runners
• Added the option for nodes to connect to a local registry for images. This registry will sit on the E2E host runners, and act as a cache for images pulled from dockerhub. This avoids problems with too many image pulls as unregistered requests are limited to 100 images per 6 hours.

Types of Changes

Verification

All CI and E2E tests still pass.

Linked Issues

TBD

User-Facing Change

Further Comments

[codecov-commenter]

Codecov Report

Base: 9.72% // Head: 9.72% // No change to project coverage 👍

Coverage data is based on head (a060805) compared to base (26e9405).
Patch has no changes to coverable lines.

Additional details and impacted files

@@          Coverage Diff           @@
##           master   #6131   +/-   ##
======================================
  Coverage    9.72%   9.72%           
======================================
  Files         138     138           
  Lines       10057   10057           
======================================
  Hits          978     978           
  Misses       8880    8880           
  Partials      199     199           

Flag	Coverage Δ
unittests	9.72% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[brandond]

Added option to register a secret for dockerhub pulls. On runners with the same IP, this was causing problems as unregistered requests are limited to 100 images per 6 hours. Adding a login from dockerhub increases that to 200 images per 6 hours and prevents failure we were seeing on runners when attempting to pull down local-path-provisioner images.

The Drone tests will start and use a local image mirror if ENABLE_REGISTRY is set; you might look at making use of this pattern for the E2E tests so that each VM doesn't have to pull all of its images directly from Docker Hub.

[dereknola]

@brandond This idea of using a cache is a much better solution. I have removed the docker login secret and moved to a docker registry running on the host.