Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Refactored acceptance test automation #2476

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

stevehipwell
Copy link
Contributor

@stevehipwell stevehipwell commented Nov 25, 2024

Resolves #2425

Depends on #2511
Depends on #2501
Depends on #2502
Depends on #2506
Depends on #2507
Depends on #2509


Before the change?

  • The acceptance tests are complicated and don't all pass
  • The acceptance tests can't be run on a PR.

After the change?

  • Acceptance tests have been refactored
    • Fewer tests
    • Simpler test structure
    • Test env setup has been made more granular and documented
  • Acceptance tests have been tested
    • Enterprise tests and external user tests have been skipped

Outstanding Tasks

  • Add acctest repo label
  • Configure non-anonymous test environment
    • Test user 1
      • Token
    • Free org (owned by test user 1)
      • Repo
      • Template repo
      • App installation
    • Test user 2
      • Member of free org
    • External test user 1
      • Token
    • External test user 2

Optional Tasks

  • Configure non-anonymous test environment
    • Team org (owned by test user 1)
      • Membership for test user 2
      • Repo
      • Template repo
      • App installation
    • Enterprise (owned by test user 1)
      • Membership for test user 2
      • Enterprise org
        • Membership for test user 2
        • Repo
        • Template repo
        • App installation
  • Add dotcom repo environment?
  • GHES?
    • Add ghes repo environment

Pull request checklist

  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been reviewed and added / updated if needed (for bug fixes / features)

Does this introduce a breaking change?

Please see our docs on breaking changes to help!

  • Yes
  • No

@stevehipwell stevehipwell force-pushed the ci-integration-tests branch 4 times, most recently from a52984b to 1a83288 Compare November 25, 2024 17:51
@stevehipwell stevehipwell force-pushed the ci-integration-tests branch 13 times, most recently from 91f4e82 to 4f20cb2 Compare November 26, 2024 22:08
@stevehipwell stevehipwell force-pushed the ci-integration-tests branch 4 times, most recently from 3e6a3e0 to 8255088 Compare November 27, 2024 13:07
@stevehipwell
Copy link
Contributor Author

@kfcampbell I've got the anonymous tests running correctly after refactoring the acceptance tests and defining all of the inputs and when they should run. I'm now at the point where I need access to test users and organisations, I've updated the description above to cover what's needed.

@stevehipwell stevehipwell force-pushed the ci-integration-tests branch 2 times, most recently from f8d8106 to c8a6cc3 Compare December 2, 2024 16:39
@stevehipwell
Copy link
Contributor Author

@kfcampbell @nickfloyd I've got all of the acceptance tests running locally that don't need paid functionality or additional users, which is most of them. We're now just waiting on some maintainer input.

@stevehipwell
Copy link
Contributor Author

This PR now depends on #2511 to have been merged first so that the tests can be run from this PR. This is still waiting on some repo and external setup tasks, but I've reduced the required list to be "good enough".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[MAINT]: Add GitHub Actions workflow for integration testing
1 participant