Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Testing framework for powershell scripts #4882

Open
sima-zhu opened this issue Dec 6, 2022 · 6 comments
Open

Testing framework for powershell scripts #4882

sima-zhu opened this issue Dec 6, 2022 · 6 comments
Assignees
Labels
Central-EngSys This issue is owned by the Engineering System team.

Comments

@sima-zhu
Copy link
Contributor

sima-zhu commented Dec 6, 2022

Background

No test coverage for our powershell scripts is long existing issue. We currently work on adding pester tests and testing in local. Several drawbacks for the local tests:

  1. It is not a convention for team member to add test coverage for everything they added.
  2. Forgot running the tests in local for new changes might introduce errors in tests, then the efforts of writing tests will be in vain.

Goals

We can have DevOps pipeline triggered every time we change the scripts.
The tests coverage plan:

  1. We can cover modularized function tests first.
  2. Better to have integration tests on single script file.
  3. Have integration tests across multiple files.

Proposals

Running tests in pipeline would be the best choices.

Proposal 1(Optimal)

Test structure
image
Triggers
Every time we touch scripts under eng/, the devOps testing pipeline would run.

Proposal 2

Test structure
image

Triggers
Every time we touch single scripts under eng/, the individual testing pipeline would run. If pipeline doesn't exist, then we use pipeline-generation create one.

@ghost ghost added the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Dec 6, 2022
@sima-zhu sima-zhu self-assigned this Dec 6, 2022
@sima-zhu sima-zhu added the Central-EngSys This issue is owned by the Engineering System team. label Dec 6, 2022
@sima-zhu
Copy link
Contributor Author

sima-zhu commented Dec 6, 2022

@weshaggard @benbp @danieljurek The proposals are currently under review and open to any suggestions. Could you please help brainstorm on a better plan?

@benbp
Copy link
Member

benbp commented Dec 6, 2022

We already have pester tests in the non-common eng directories of the language repositories as well, right? It would be nice if the pipeline testing steps are modular enough that we could add them into language pipelines easily (I'm not sure the right place yet).

@benbp
Copy link
Member

benbp commented Dec 6, 2022

My initial inclination is to start with just one pipeline that discovers all pester tests in the repository, as opposed to more directory specific tests, both for simplicity and because we have a lot of cross-directory script dependencies that aren't easily tracked.

@sima-zhu sima-zhu removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Dec 6, 2022
@weshaggard
Copy link
Member

My initial inclination is to start with just one pipeline that discovers all pester tests in the repository, as opposed to more directory specific tests

I agree with one pipeline but I would still stick to limiting to a given area not looking for all pester tests across the entire repo. For that reason I sort of lean towards proposal 1 with a scripts-tests directory that we scope the pester tests that we want to run to.

@sima-zhu
Copy link
Contributor Author

@sima-zhu
Copy link
Contributor Author

Next step is to have eng/scripts-tests pipelines for tools, net repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Central-EngSys This issue is owned by the Engineering System team.
Projects
Status: 📋 Backlog
Development

No branches or pull requests

4 participants