Provide an (at least temporary) non-global ManagedHandler opt-in mechanism

[stephentoub]

Right now the only way to opt-in to using ManagedHandler is via an environment variable. That requires a process-wide setting which makes it impossible to create an individual HttpClient{Handler} that uses ManagedHandler; another thread could be instantiating one at the same time and would pick up the setting. This is why we had to disable parallel execution of our System.Net.Http test suite.

We still haven't yet figured out the final model via which we'll enable usage of ManagedHandler. For now, though, I'm adding a thread-local switch, to allow opting in a single HttpClientHandler construction. This makes it possible to re-enable our test parallelism, which this PR also does (and in the process drops outerloop execution time by 3x on my machine), but it also enables us to make progress on https://github.com/dotnet/corefx/issues/23152, which is blocked by having a global-only setting.

cc: @geoffkizer, @davidsh, @wfurt, @Priya91

[davidsh]

LGTM

[wfurt]

looks good to me. It would be nice to have some API to enable managed handler via API call.

[davidsh]

I assume you'll run Outerloop on all this before merge?

[stephentoub]

I assume you'll run Outerloop on all this before merge?

Yes. I was waiting for the other legs to pass before doing so.

[stephentoub]

@dotnet-bot test Outerloop Linux x64 Debug Build please
@dotnet-bot test Outerloop Windows x64 Debug Build please
@dotnet-bot test Outerloop OSX x64 Debug Build please

[stephentoub]

@dotnet/dnceng, can you help me to understand why the Outerloop Windows leg here is showing as failing?
https://ci3.dot.net/job/dotnet_corefx/job/master/job/windows-TGroup_netcoreapp+CGroup_Debug+AGroup_x64+TestOuter_true_prtest/70/
When I click through to it, the only failures shown are on macOS. What am I missing?

[stephentoub]

The macOS leg is hitting "Too many open files". @dotnet/dnceng, what is the ulimit for open file handles on the macOS machines used in these test runs? This change re-enables parallelism of tests, so it's possible various tests running at the same time are causing too many handles to be opened, but that should really only happen if the limit is set really low (or if there's a bug in the tests that's resulting in handles getting leaked incorrectly).

[mmitche]

@stephentoub These are helix machines, so maybe the limits are set lower?
@MattGal May have an opinion on this. I think he was looking to have the tests set up properly to invoke the process with a higher files limit rather than changing the OS globally.

[wfurt]

There are at least two limits - file descriptors per process and also system wide limit.
This one seems like the first case.

[MattGal]

Please see https://github.com/dotnet/corefx/blob/4454822040671e10db4fc7e5670bbe8bc0342b0c/src/System.Console/tests/Performance/System.Console.Performance.Tests.csproj#L27-29 . This approach allows you to specify an increased limit for a single test, which prevents missing problems by artificially setting this very high system-wide.

[stephentoub]

@MattGal, what is the default on macOS in helix when this isn't done?

Also, any answer to my Windows question?

Thanks.

[MattGal]

Disclaimer: Macs are manually set up and may vary slightly
Ulimit -a on several machines from our OSX 10.12 queue shows the following:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

The windows problem is an issue with the Jenkins plugin trying to talk to Helix. No tests actually failed, as you note, but the plugin got an HTTP 502 ("502 - Web server received an invalid response while acting as a gateway or proxy server. There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server." ) and translated this to some sort of failure.

@ChadNedzlek may have some thoughts on this (from the Helix API perspective) but he's on vacation right this moment. In the mean time we should work to handle this response type (as it does appear to be transient) from the Jenkins plugin.

[danmoseley]

Hopefully dotnet/coreclr#14054 will fix this at least for .NET Core code.

[stephentoub]

Thanks, @MattGal.

This:

open files                      (-n) 256

is kind of ridiculously low for a runtime/libraries focused heavily on services. We previously saw tons of failures due to this macOS default in Jenkins; I'm surprised we haven't seen more in Helix. I will increase the value for this specific test project in order to unblock my change, but I think we should increase this limit across the board. I don't know of any case where having this low limit has helped us to catch bugs, and I expect we'll just find ourselves getting random failures here and there due to this, at which point we'd just increase the limit in that project, and eventually most projects will have the setting.

[stephentoub]

@dotnet-bot test Outerloop OSX x64 Debug Build please

[stephentoub]

Fixes https://github.com/dotnet/corefx/issues/24190 (which coincidentally was opened after this PR 😄)

[MattGal]

@stephentoub, as to that value being "ridiculously low for a runtime/libraries focused heavily on services", I totally agree but that's the default... we didn't set it there, we simply took macs and used them. We've done nothing to these machines to change that.

Thus, unless there's something I'm missing this is exactly what newbie OSX developers experience; if the change isn't made inside the test's execution script, then every new mac developer will have to learn this workaround before they can reliably run and pass the tests. It's one of those things like needing to open a range of firewall ports on Fedora; it makes the new-user experience not the greatest. As such, given we already have cheap and easy mechanisms in place to make the tests resilient in this way I strongly prefer this mechanism over a permanent machine change that the end user will have to just figure out via reading forum posts and experimentation. Feel free to open a core-eng issue if you feel otherwise and we can discuss it in the next triage session.

[stephentoub]

I'd be fine if the answer is we enable this for all tests by putting 2bce06e#diff-03ce3895f0647a98768807ca9ec6382eR134 in some top-level targets file. My goal is simply that we enable this for all tests and not just piecemeal do so ad-hoc when tests randomly fail in this project or that.