Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.

Re-partition MSBuild test groups #22885

Merged
merged 3 commits into from
Feb 28, 2019
Merged

Re-partition MSBuild test groups #22885

merged 3 commits into from
Feb 28, 2019

Conversation

echesakov
Copy link

@echesakov echesakov commented Feb 27, 2019

This changes a tests paritioning scheme of #17161 in several different ways:

  1. Make the test groups are of equal (to some extent) sizes keeping the number of tests in one group less than 1000;
  2. As a resulf of 1) this increases a number of groups needed for Pri0 tests to 3 and descreases a number of groups for Pri1 tests to 10;
  3. This also changes a way of tests partitioning scheme is defined - instead of explicitly specifying MSBuild Include-patterns this defines a boundaries between test groups in a form of _StartsWith ItemGroup. Then I use StringComparer.Ordinal to pick tests that belong to a particular group.

All of this was done to mitigate https://github.com/dotnet/coreclr/issues/20236 which I believe caused by OOM due to enormous number of projects loaded during build-tests (same reasoning as in #17161).

Also fixes https://github.com/dotnet/coreclr/issues/22627 and should prevent from such issues in the future.

I want to mention that I was considering using automatic scheme before converging back to the manual one.
There are several problem with automatic partitioning that in theory can be solved but would take too much effort to do this and probably can be postponed:

  1. The priority of a test (and whether the test is going to be built) is not known during dirs.proj time - we need to import a test first to figure this out;
  2. If we ignore 1) and make an assumption that pri0 and pri1 tests are uniformly distributed than we can do different tricks with partitioning scheme. For example, Group n can composed by taking every n-th test. However, I don't know about a way to do this in MSBuild other than implementing logic in C# MSBuild task. It could be implemented using regexps (and I have a prototype doing this) but the code would look so cryptic that I decided not do this;
  3. The proper way of doing automatic partitioning should be based on some hash function (which is naturally uniformly distributed). As an example, the hash can be computed based on a test file name as input. However, the hash function directly available in MSBuild is String.GetHashCode is not stable - meaning the distribution would change from run to run adding non-determinism into build system.

@dotnet/jit-contrib

@echesakov
Copy link
Author

@AaronRobinsonMSFT Can you please take a look at the MSBuild changes ?

Copy link

@sdmaclea sdmaclea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the new approach. It seems like it will be less likely to lead to missing tests.

Only did a quick scan of the change...

Copy link
Member

@jkoritzinsky jkoritzinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MSBuild logic looks good to me.

tests/src/dirs.proj Outdated Show resolved Hide resolved
tests/src/dirs.proj Show resolved Hide resolved
@echesakov
Copy link
Author

/azp run coreclr-outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@echesakov
Copy link
Author

Both coreclr-ci and coreclr-outerloop have succeeded.
Ignoring Jenkins infra failures.

@echesakov echesakov merged commit 39b643f into dotnet:master Feb 28, 2019
@echesakov echesakov deleted the RePartitionMSBuildTestSlices branch February 28, 2019 17:55
picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
This changes a tests partitioning scheme of coreclr tests in several different ways:

1. Make the test groups are of equal (to some extent) sizes keeping the number of tests in one group less than 1000;

2. As a resulf of 1) this increases a number of groups needed for Pri0 tests to 3 and decreases a number of groups for Pri1 tests to 10;

3. This also changes a way of tests partitioning scheme is defined - instead of explicitly specifying MSBuild Include-patterns this defines a boundaries between test groups in a form of _StartsWith ItemGroup. Then I use StringComparer.Ordinal to pick tests that belong to a particular group.


Commit migrated from dotnet/coreclr@39b643f
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test build on linux misses some test directories
5 participants