e2e test with installer

[anusha94]

What this PR does / why we need it:

• removed the skip-installation flag from e2e test
• updated test/e2e/BYOHDockerfile to exclude k8s components
• added a few additional required packages (ufw and dbus) to the Dockerfile
• added a new dev-dockerfile to continue having the same experience in
  getting started and local-dev guides

Successful e2e run: https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost/runs/4800081237?check_suite_focus=true

Notes for Reviewer:
Adding additional Dockerfile so that the current getting started guides are not affected by this change. This PR's aim is to enable running e2e tests in GH Actions without skipping the installer workflow. Currently, the getting started guides skip this step and we want to continue doing that until we figure out a better way to not modify host config or some other alternative.

[anusha94]

@jamiemonserrate In the e2e test, we don't have to start the agent with sudo because the container itself it started with privileged = true

cluster-api-provider-bringyourownhost/test/e2e/e2e_docker_helper.go

Line 141 in 26b4bc2

&container.HostConfig{Privileged: true,

@jamiemonserrate @dharmjit
wdyt of the contents of BYOHDockerfile? Currently I've removed containerd and other k8s bits. Anything else that the installer is taking care of and shouldn't be present in the Dockerfile?

[jamiemonserrate]

Reviewed. This LGTM.

We still have the issue that the tests will change kernel settings for users who attempt to run these tests locally. But lets fix that in a subsequent PR.

[dharmjit]

This looks good to me.

Nitpick only, Is there some other dir to keep BYOHDockerFileDev. Do you thinkBYOHDevDockerFile is a better name.

[anusha94]

This looks good to me.

Nitpick only, Is there some other dir to keep BYOHDockerFileDev. Do you thinkBYOHDevDockerFile is a better name.

Yea I can change the filename. Where do you think we can keep this file? I kept in the docs folder as it is being used there

[dharmjit]

Where do you think we can keep this file?

My initial thoughts were docs should only contain documentation and not code. Do you think root dir could be an alternate?

[anusha94]

My initial thoughts were docs should only contain documentation and not code. Do you think root dir could be an alternate?

Will do this separately as part of a different PR. Merged this so that other PRs can benefit from running a full e2e test.