Draft: Windows: Diagnose msys openssl fork issue

[sunjayBhatia]

DO NOT MERGE

See failed build: https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/52196/logs/126

error:

e is 65537 (0x010001)
      1 [main] openssl 125 child_info_fork::abort: \??\C:\tools\msys64\usr\bin\msys-crypto-1.1.dll: Loaded to different address: parent(0x58B980000) != child(0x400000)
Can't open good_cert.pem for reading, No such file or directory
34359738384:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('good_cert.pem','r')
34359738384:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
unable to load certificate
Failed bash -c "PYTHONPATH=$(dirname C:/botcode/w/bazel-out/x64_windows-opt/bin/test/extensions/transport_sockets/tls/handshaker_test.exe.runfiles/envoy/test/extensions/transport_sockets/tls/ocsp/gen_unittest_ocsp_data.sh) C:/botcode/w/bazel-out/x64_windows-opt/bin/test/extensions/transport_sockets/tls/handshaker_test.exe.runfiles/envoy/test/extensions/transport_sockets/tls/ocsp/gen_unittest_ocsp_data.sh "
[2020-09-25 18:47:59.960][2168][critical][assert] [test/test_common/environment.cc:413] assert failure: false.

---

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]

[sunjayBhatia]

looks like disabling usosvc fixes the issue (update orchestrator service)

this PR is required to fix this defect: envoyproxy/envoy-build-tools#100

[sunjayBhatia]

ran a test that utilizes msys2 openssl: https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/52525/logs/16

succeeds when usosvc is stopped

[sunjayBhatia]

ran a test that utilizes msys2 openssl: https://dev.azure.com/cncf/4684fb3d-0389-4e0b-8251-221942316e06/_apis/build/builds/52525/logs/16

succeeds when usosvc is stopped

Turns out this was wrong, the invocation of powershell was enough to slow down the script and make the flake occurrence less frequent but it still exists

[sunjayBhatia]

It looks like possibly an issue as described here, not entirely sure https://cygwin.com/faq.html#faq.using.bloda

[sunjayBhatia]

Running msys2 autorebase.bat when building the build container image does not help

This flake only occurs on RBE, we have not been able to repro it locally on our dev VM or in a basic GCP Windows server VM

[sunjayBhatia]

Also attempting to build and use the bssl tool as an openssl tool equivalent from Boring SSL will not suffice, it has a much more limited set of capabilities as far as I can tell

[wrowe]

Thanks for all the research. With the complexity of TSL1.3 features and QUIC, the most flexible build host tool is probably needed. What are your thoughts, should we always pick up a modern openssl and build the same tool for every host, or deploy the win32 flavor of openssl at provisioning time?

[sunjayBhatia]

Thanks for all the research. With the complexity of TSL1.3 features and QUIC, the most flexible build host tool is probably needed. What are your thoughts, should we always pick up a modern openssl and build the same tool for every host, or deploy the win32 flavor of openssl at provisioning time?

Given that OpenSSL isn't bazelified already, adding another foreign_cc build to get the openssl.exe tool may not get accepted? I tried mirroring what chocolatey does and installing with the 64 bit OpenSSL installer from here, and it works well, I'll run the Envoy tests building a build container with that. We could maybe even build the openssl.exe tool in the build container build process, but using the installer is quicker.

[sunjayBhatia]

Thanks for all the research. With the complexity of TSL1.3 features and QUIC, the most flexible build host tool is probably needed. What are your thoughts, should we always pick up a modern openssl and build the same tool for every host, or deploy the win32 flavor of openssl at provisioning time?

Given that OpenSSL isn't bazelified already, adding another foreign_cc build to get the openssl.exe tool may not get accepted? I tried mirroring what chocolatey does and installing with the 64 bit OpenSSL installer from here, and it works well, I'll run the Envoy tests building a build container with that. We could maybe even build the openssl.exe tool in the build container build process, but using the installer is quicker.

Installing standalone OpenSSL worked, see envoyproxy/envoy-build-tools#107 for the build container update coming soon

[sunjayBhatia]

Fixed by #13702