Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm] Add disabled tests to ProjectExclusions #65877

Merged
merged 2 commits into from
Feb 25, 2022

Conversation

elinor-fung
Copy link
Member

Microsoft.Extensions.Caching.Memory.Tests and System.ComponentModel.Composition.Tests are already disabled via assembly-level attributes in source.

[assembly: ActiveIssue("https://github.com/dotnet/runtime/issues/35970", TestRuntimes.Mono)]

[assembly: ActiveIssue("https://github.com/mono/mono/issues/16417", TestRuntimes.Mono)] // flaky tests

They are actually disabled for all mono, but we don't have a mechanism in place of avoiding the build, work item queuing, test discovery, result upload, and such for desktop Mono (since we build libraries tests once and use them for coreclr and non-device mono test runs). The wasm ones were the most time-intensive due to AOT-ing on the helix machine.

Based on the below query, in the past two days, running those two tests suites (neither of which actually run any tests) for wasm has taken about 6.75 days of various machines' time.

WorkItems
| where Started > ago(2d)
| join kind=inner Jobs on JobId
| where FriendlyName endswith "System.ComponentModel.Composition.Tests"
  or FriendlyName endswith "Microsoft.Extensions.Caching.Memory.Tests"
| project
  FriendlyName,
  Queued, Started, Finished,
  Duration = Finished - Started,
  QueueName,
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Runtime = tostring(parse_json(Properties).runtimeFlavor),
  ConsoleUri,
  Source
| where Runtime == "mono"
| where PhaseName contains "Browser"
| summarize sum(Duration) 

cc @dotnet/runtime-infrastructure

@ghost
Copy link

ghost commented Feb 25, 2022

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

Microsoft.Extensions.Caching.Memory.Tests and System.ComponentModel.Composition.Tests are already disabled via assembly-level attributes in source.

[assembly: ActiveIssue("https://github.com/dotnet/runtime/issues/35970", TestRuntimes.Mono)]

[assembly: ActiveIssue("https://github.com/mono/mono/issues/16417", TestRuntimes.Mono)] // flaky tests

They are actually disabled for all mono, but we don't have a mechanism in place of avoiding the build, work item queuing, test discovery, result upload, and such for desktop Mono (since we build libraries tests once and use them for coreclr and non-device mono test runs). The wasm ones were the most time-intensive due to AOT-ing on the helix machine.

Based on the below query, in the past two days, running those two tests suites (neither of which actually run any tests) for wasm has taken about 6.75 days of various machines' time.

WorkItems
| where Started > ago(2d)
| join kind=inner Jobs on JobId
| where FriendlyName endswith "System.ComponentModel.Composition.Tests"
  or FriendlyName endswith "Microsoft.Extensions.Caching.Memory.Tests"
| project
  FriendlyName,
  Queued, Started, Finished,
  Duration = Finished - Started,
  QueueName,
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Runtime = tostring(parse_json(Properties).runtimeFlavor),
  ConsoleUri,
  Source
| where Runtime == "mono"
| where PhaseName contains "Browser"
| summarize sum(Duration) 

cc @dotnet/runtime-infrastructure

Author: elinor-fung
Assignees: elinor-fung
Labels:

area-Infrastructure-libraries

Milestone: -

@radical
Copy link
Member

radical commented Feb 25, 2022

We should probably then drop the ActiveIssue. That way somebody removing the project exclusions won't be mistaken by "no CI failures".

cc @lewing

@radical radical added the arch-wasm WebAssembly architecture label Feb 25, 2022
@ghost
Copy link

ghost commented Feb 25, 2022

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Microsoft.Extensions.Caching.Memory.Tests and System.ComponentModel.Composition.Tests are already disabled via assembly-level attributes in source.

[assembly: ActiveIssue("https://github.com/dotnet/runtime/issues/35970", TestRuntimes.Mono)]

[assembly: ActiveIssue("https://github.com/mono/mono/issues/16417", TestRuntimes.Mono)] // flaky tests

They are actually disabled for all mono, but we don't have a mechanism in place of avoiding the build, work item queuing, test discovery, result upload, and such for desktop Mono (since we build libraries tests once and use them for coreclr and non-device mono test runs). The wasm ones were the most time-intensive due to AOT-ing on the helix machine.

Based on the below query, in the past two days, running those two tests suites (neither of which actually run any tests) for wasm has taken about 6.75 days of various machines' time.

WorkItems
| where Started > ago(2d)
| join kind=inner Jobs on JobId
| where FriendlyName endswith "System.ComponentModel.Composition.Tests"
  or FriendlyName endswith "Microsoft.Extensions.Caching.Memory.Tests"
| project
  FriendlyName,
  Queued, Started, Finished,
  Duration = Finished - Started,
  QueueName,
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Runtime = tostring(parse_json(Properties).runtimeFlavor),
  ConsoleUri,
  Source
| where Runtime == "mono"
| where PhaseName contains "Browser"
| summarize sum(Duration) 

cc @dotnet/runtime-infrastructure

Author: elinor-fung
Assignees: elinor-fung
Labels:

arch-wasm, area-Infrastructure-libraries

Milestone: -

@elinor-fung
Copy link
Member Author

The ActiveIssue is still needed for test runs on Mono. Is your suggestion to condition it to be for Mono, but not Browser? That seems kinda weird to me.

@radical
Copy link
Member

radical commented Feb 25, 2022

Oh, sorry, I misunderstood! nit: add a comment in the proj file that they are also disabled in the source.

@elinor-fung elinor-fung merged commit 95f7f7a into dotnet:main Feb 25, 2022
@elinor-fung elinor-fung deleted the testsProjectExclusion branch February 25, 2022 15:59
@ghost ghost locked as resolved and limited conversation to collaborators Mar 27, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants