Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exclude Paket from exploration tests for now #6283

Merged
merged 1 commit into from
Nov 14, 2024
Merged

Exclude Paket from exploration tests for now #6283

merged 1 commit into from
Nov 14, 2024

Conversation

andrewlock
Copy link
Member

Summary of changes

Excludes Paket from exploration tests

Reason for change

Paket apparently doesn't work with .NET 9, it fails to restore:

[ERR] SetUpExplorationTest: D:\a\1\s\exploration-tests\paket\.paket\Paket.Restore.targets(171,5): error MSB3073: The command ""D:\a\1\s\exploration-tests\paket\.paket\paket.exe" restore" exited with code -1. [D:\a\1\s\exploration-tests\paket\src\Paket.Core\Paket.Core.fsproj]
[ERR] SetUpExplorationTest: D:\a\1\s\exploration-tests\paket\.paket\Paket.Restore.targets(171,3): error MSB3073: The command ""D:\a\1\s\exploration-tests\paket\.paket\paket.exe" restore" exited with code -1. [D:\a\1\s\exploration-tests\paket\src\Paket\Paket.fsproj]

Implementation details

Just exclude Paket from the tests for now

Test coverage

Less

Other details

I didn't notice that this was a genuine failure in the .NET 9 PR because these fail so much that I never pay them any attention 🙈

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Nov 14, 2024
@andrewlock andrewlock requested a review from a team as a code owner November 14, 2024 10:45
@andrewlock andrewlock force-pushed the andrew/dotnet-9-exploration-tests branch from 0b07a9f to 9f672ea Compare November 14, 2024 10:48
@datadog-ddstaging
Copy link

datadog-ddstaging bot commented Nov 14, 2024 •
edited
Loading

Datadog Report

Branch report: andrew/dotnet-9-exploration-tests
Commit report: 9f672ea
Test service: dd-trace-dotnet

✅ 0 Failed, 468817 Passed, 3606 Skipped, 32h 20m 17.6s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

  • WorkingWithContinuousProfiler - Datadog.Trace.Tools.dd_dotnet.ArtifactTests.Checks.ProcessBasicChecksTests - Last Failure

    Expand for error 
     
 
 With StandardOutput:
 Running checks on process 2676 Process name: Samples.Console  ---- STARTING TRACER SETUP CHECKS ----- Target process is running with .NET Framework 1. Checking Modules Needed so the Tracer Loads:  [SUCCESS]: The native library version 3.6.0.0 is loaded into the process.  [WARNING]: Found multiple instances of Datadog.Trace.dll in the target process. Detected versions: 3.6.0.0  2. Checking DD_DOTNET_TRACER_HOME and related configuration value:  [SUCCESS]: DD_DOTNET_TRACER_HOME is set to 'D:\a\1\s\shared\bin\monitoring-home' and the directory was found correctly. 3. Checking COR_PROFILER_PATH and related configuration value:  [SUCCESS]: The environment variable COR_PROFILER_PATH is set to the correct value of D:\a\1\s\shared\bin\monitoring-home\win-x64\Datadog.Trace.ClrProfiler.Native.dll . 4. Checking COR_PROFILER and related configuration value:  [SUCCESS]: The environment variable COR_PROFILER is set to the correct value of {846F5F1C-F9AE-4B07-969E-05C26BC060D8}. 5. Checking COR_ENABLE_PROFILING and related configuration value:  [SUCCESS]: The environment variable COR_ENABLE_PROFILING is set to the correct value of 1.  ---- CONFIGURATION CHECKS ----- 1. Checking if tracing is disabled using DD_TRACE_ENABLED.  [INFO]: DD_TRACE_ENABLED is not set, the default value is true. 2. Checking if profiling is enabled using DD_PROFILING_ENABLED.  [SUCCESS]: DD_PROFILING_ENABLED is set.  ---- DATADOG AGENT CHECKS ----- Detected agent url: http://127.0.0.1:52089/. Note: this url may be incorrect if you configured the application through a configuration file. Connecting to Agent at endpoint http://127.0.0.1:52089/ using HTTP  [WARNING]: Could not detect the agent version. It may be running with a version older than 7.27.0.  [SUCCESS]: No issue found with the target process. 
 With ErrorOutput:
 
 With ExitCode:
 0acer version 3.6.0.0 is loaded into the process."}.

@andrewlock
Copy link
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6283) - mean (72ms)  : 64, 80
     .   : milestone, 72,
    master - mean (72ms)  : 63, 82
     .   : milestone, 72,

    section CallTarget+Inlining+NGEN
    This PR (6283) - mean (1,108ms)  : 1089, 1127
     .   : milestone, 1108,
    master - mean (1,110ms)  : 1091, 1129
     .   : milestone, 1110,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6283) - mean (108ms)  : 106, 110
     .   : milestone, 108,
    master - mean (109ms)  : 106, 111
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6283) - mean (770ms)  : 753, 787
     .   : milestone, 770,
    master - mean (765ms)  : 747, 783
     .   : milestone, 765,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6283) - mean (92ms)  : 90, 94
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (6283) - mean (725ms)  : 709, 741
     .   : milestone, 725,
    master - mean (726ms)  : 709, 742
     .   : milestone, 726,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6283) - mean (190ms)  : 187, 193
     .   : milestone, 190,
    master - mean (191ms)  : 184, 199
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6283) - mean (1,230ms)  : 1207, 1253
     .   : milestone, 1230,
    master - mean (1,228ms)  : 1207, 1249
     .   : milestone, 1228,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6283) - mean (276ms)  : 271, 280
     .   : milestone, 276,
    master - mean (277ms)  : 273, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6283) - mean (950ms)  : 928, 972
     .   : milestone, 950,
    master - mean (943ms)  : 925, 961
     .   : milestone, 943,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6283) - mean (265ms)  : 261, 270
     .   : milestone, 265,
    master - mean (266ms)  : 262, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (6283) - mean (927ms)  : 909, 944
     .   : milestone, 927,
    master - mean (931ms)  : 914, 947
     .   : milestone, 931,
Loading

@andrewlock
Copy link
Member Author

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6283) (11.236M)   : 0, 11235535
    master (11.174M)   : 0, 11173945
    benchmarks/2.9.0 (11.033M)   : 0, 11032866

    section Automatic
    This PR (6283) (7.307M)   : 0, 7307125
    master (7.312M)   : 0, 7312280
    benchmarks/2.9.0 (7.786M)   : 0, 7785853

    section Trace stats
    master (7.553M)   : 0, 7553254

    section Manual
    master (11.102M)   : 0, 11101684

    section Manual + Automatic
    This PR (6283) (6.761M)   : 0, 6761489
    master (6.740M)   : 0, 6739647

    section DD_TRACE_ENABLED=0
    master (10.282M)   : 0, 10282028
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6283) (9.601M)   : 0, 9601402
    master (9.530M)   : 0, 9530431
    benchmarks/2.9.0 (9.495M)   : 0, 9494821

    section Automatic
    This PR (6283) (6.343M)   : 0, 6343380
    master (6.395M)   : 0, 6394628

    section Trace stats
    master (6.680M)   : 0, 6679934

    section Manual
    master (9.631M)   : 0, 9630609

    section Manual + Automatic
    This PR (6283) (5.842M)   : 0, 5841717
    master (5.995M)   : 0, 5995215

    section DD_TRACE_ENABLED=0
    master (8.922M)   : 0, 8921645
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6283) (9.884M)   : 0, 9883684
    master (9.758M)   : 0, 9758058
    benchmarks/2.9.0 (10.020M)   : 0, 10019592

    section Automatic
    This PR (6283) (6.279M)   : 0, 6278695
    master (6.513M)   : 0, 6512759
    benchmarks/2.9.0 (7.255M)   : 0, 7255257

    section Trace stats
    master (7.074M)   : 0, 7073637

    section Manual
    master (9.827M)   : 0, 9827350

    section Manual + Automatic
    This PR (6283) (5.901M)   : 0, 5900727
    master (5.944M)   : 0, 5943752

    section DD_TRACE_ENABLED=0
    master (9.348M)   : 0, 9348488
Loading

@andrewlock andrewlock merged commit 9d4a0fa into master Nov 14, 2024
80 of 82 checks passed
@andrewlock andrewlock deleted the andrew/dotnet-9-exploration-tests branch November 14, 2024 13:37
@github-actions github-actions bot added this to the vNext-v3 milestone Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vandonr vandonr vandonr approved these changes

Assignees
No one assigned
Labels
area:builds project files, build scripts, pipelines, versioning, releases, packages
Projects
None yet
Milestone
3.6.0
Development

Successfully merging this pull request may close these issues.
2 participants
@andrewlock @vandonr