Figure out a regression test plan

[tianyin]

It seems that the github tests are flaky and always timeout, and we are mostly relying on cloudlab tests?

@ThrivikramanV Could you figure out the right practice to do regression tests, given that we will have more folks working on the project like @MarkintoshZ ?

[ThrivikramanV]

Sure, on this.

[tianyin]

Thanks @ThrivikramanV ! It's great that we can resolve this long-lasting issue!

[tylergu]

Let me use this issue to keep track of the planned actions so that we don't forget them

• Have a Github action that randomly select one bug to reproduce. This action can run quickly, and is run for each PR. (Modified e2e regression tests to reproduce only one bug #275)

• Have a weekly action to reproduce all bugs. For this, we need to recruit one idle lab machine to use as the Github self-hosted runner. I think PT2 and PT3 are mostly idle these days.

• Better organize the unittests and e2e tests. We want to separate unittests and e2e tests, because we want separate github actions for them. There are two ways to go with this:

  • Putting unittests and e2e tests in separate directory, so that the github actions can just invoke pytest on different directories to separate unittest and e2e tests.
  • Using pytest marker to annotate each test, and the github action invokes the tests using the markers.

  The first option is probably better, and we are already in a good shape for that (our unittests are in the source code directories, and e2e tests are in the test directory)