Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a BDD Test Automation Framework with ginkgo #552

Merged
merged 3 commits into from
Feb 5, 2019
Merged

Create a BDD Test Automation Framework with ginkgo #552

merged 3 commits into from
Feb 5, 2019

Conversation

alejovicu
Copy link
Contributor

@alejovicu alejovicu commented Oct 26, 2018

@openshift-ci-robot openshift-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 26, 2018
@wking
Copy link
Member

wking commented Oct 26, 2018

Do we need this? For the libvirt tests, I was hoping to run the usual origin e2e suite. And if you're only concerned about exercising create install-config, a simpler approach would be to add a cmd/openshift-install/create_test.go for our existing Go unit job.

@alejovicu
Copy link
Contributor Author

Hit @wking
This framework should cover the entire application.
At this moment is just the setup of the framework and the test suite for "create install-config" command. but It will increase adding the test suites for "create ignition-configs, create manifests, create cluster, destroy-cluster...". When the cluster is deployed, the usual origin e2e suite will be executed to certify the cluster health.

Now, in the test workflow, the idea is to run the config tests before the cluster health tests because some issues can be found without the cluster deploy saving some time.

@alejovicu alejovicu changed the title Add test suite for create install config command [WIP] Add test suite for create install config command Nov 13, 2018
@openshift-ci-robot openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 13, 2018
@alejovicu alejovicu changed the title [WIP] Add test suite for create install config command Create a BDD test automation framework using ginkgo Dec 6, 2018
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2018
@alejovicu alejovicu changed the title Create a BDD test automation framework using ginkgo Create a BDD Test Automation Framework with ginkgo Dec 6, 2018
Copy link
Contributor

@crawford crawford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, this looks good to me. Just a few minor requests.

tests/bdd-smoke/Gopkg.toml Show resolved Hide resolved
tests/bdd-smoke/utils/assertion.go Outdated Show resolved Hide resolved
tests/bdd-smoke/utils/assertion.go Outdated Show resolved Hide resolved
@abhinavdahiya
Copy link
Contributor

@alejovicu

  • can we quash this into 3 commits, vendor, scripts/docs, bdd code.
  • also any reason why this has a separate vendor?

@alejovicu
Copy link
Contributor Author

Hi @abhinavdahiya,
The split in 3 commits is done.
I created this as a separate vendor because this framework uses dependencies that only apply for it, like gomega that would be useless by other modules in the installer.

hack/run-bdd-suite.sh Outdated Show resolved Hide resolved
@abhinavdahiya
Copy link
Contributor

This looks like a good first step, Thanks a lot @alejovicu :yay:
Lets iterate on this.

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 5, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, alejovicu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 5, 2019
@abhinavdahiya
Copy link
Contributor

/refresh

Copy link
Member

@wking wking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few nits inline. Nothing that needs to block a merge, but probably things that are worth fixing post-merge.

Specify("Then I expect to see the install-config.yaml file in the path ./scenario1/", func() {
Ω(openshiftInstallerPath + "/scenario1/install-config.yaml").Should(BeAnExistingFile())
})
Specify("And I expect install-config.yaml file contains the info specified in the environment variables", func() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: these aren't environment variables any more (and also in several other Specify strings in this commit).

cmd.Stderr = &errb
err := cmd.Run()
if err != nil {
fmt.Println("Error: failed running the command \"" + command + "\" with the arguments:")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this could be more robust and compact with:

fmt.Printf("Error: failed running the command %q with the arguments:", command)

although printing straight to the terminal feels strange anyway. Does that get captured somewhere? Should we be logging these instead, or attaching it to the error before passing it to log.Fatal?

var installState installStateConf
json.Unmarshal(byteValue, &installState)

AssertStringContains(installState.InstallConfig.Config.BaseDomain,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and later parts of ValidateInstallStateConfig feel like a hand-crafted special case of reflect.DeepEqual. Is it doing something else?

expectedData.ClusterName,
"*installconfig.clusterName.ClusterName not found in "+fileName)

// OPENSHIFT_INSTALL_PLATFORM is implicitly validated by next checks. If its not Libvirt platform, the Name and URI will not be accessible
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This references the environment variable dropped in 6be4c25 (#861).

@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci-robot
Copy link
Contributor

@alejovicu: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-openstack 1b092fe link /test e2e-openstack
ci/prow/e2e-libvirt 1b092fe link /test e2e-libvirt

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit c5ffe9b into openshift:master Feb 5, 2019
@alejovicu alejovicu deleted the libvirt-tests branch February 6, 2019 16:39
wking pushed a commit to eparis/installer that referenced this pull request Feb 16, 2019
From Trevor:

For regions like eu-west-3 (Paris) that don't support m4.  The only
instance type available in all AWS regions today is i3.  We prefer m4
because of the higher EBS attach limit.  But for regions that don't
support m4 we use m5.

This also adds a CI job which checks the AWS pricing API to keep our
region map up to date.

The eu-north-1 region is new as of 2018-12-12 [1], and it show up in:

  $ aws ec2 describe-regions --query 'Regions[].RegionName' --output json | jq -r 'sort[]'
  ...
  eu-central-1
  eu-north-1
  eu-west-1
  ...

The GovCloud entries didn't show up in that query (maybe they are only
listed for accounts that have access?), but they are listed on [2].

The new test is in platformtests/aws to keep it out of our default
unit tests.  Maintainers can run it manually to verify that the
default fallbacks we hard-code are still appropriate for the current
AWS offerings.  It's not in tests/aws, because 'dep ensure' seems to
ignore tests.  I think it's fine to have both platformtests and tests
in parallel, with the former holding tests who share the project
vendor/ and the latter holding tests which have a distinct vendor/.
bdd-smoke is in tests with its own vendor/ intentionally to keep
gomega and other test-specific libraries out of the project vendor/
[3].  But for these tests we're using Go's built-in test framework, so
it's less maintenance to layer our minimal additional dependencies
into the project vendor/.

[1]: https://aws.amazon.com/blogs/aws/now-open-aws-europe-stockholm-region/
[2]: https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
[3]: openshift#552 (comment)
wking pushed a commit to eparis/installer that referenced this pull request Feb 19, 2019
From Trevor:

For regions like eu-west-3 (Paris) that don't support m4.  The only
instance type available in all AWS regions today is i3.  We prefer m4
because of the higher EBS attach limit.  But for regions that don't
support m4 we use m5.

This also adds a CI job which checks the AWS pricing API to keep our
region map up to date.

The eu-north-1 region is new as of 2018-12-12 [1], and it show up in:

  $ aws ec2 describe-regions --query 'Regions[].RegionName' --output json | jq -r 'sort[]'
  ...
  eu-central-1
  eu-north-1
  eu-west-1
  ...

The GovCloud entries didn't show up in that query (maybe they are only
listed for accounts that have access?), but they are listed on [2].

The new test is in platformtests/aws to keep it out of our default
unit tests.  Maintainers can run it manually to verify that the
default fallbacks we hard-code are still appropriate for the current
AWS offerings.  It's not in tests/aws, because 'dep ensure' seems to
ignore tests.  I think it's fine to have both platformtests and tests
in parallel, with the former holding tests who share the project
vendor/ and the latter holding tests which have a distinct vendor/.
bdd-smoke is in tests with its own vendor/ intentionally to keep
gomega and other test-specific libraries out of the project vendor/
[3].  But for these tests we're using Go's built-in test framework, so
it's less maintenance to layer our minimal additional dependencies
into the project vendor/.

[1]: https://aws.amazon.com/blogs/aws/now-open-aws-europe-stockholm-region/
[2]: https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/
[3]: openshift#552 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants