Fixes dynamic loading bug

[maddeleine]

Resolved issues:

resolves #3987

Description of changes:

This fixes the crash when libs2n is dynamically loaded and also includes a test for this scenario. Technically this succeeds in both a codebuild and nix run. So I'll keep that around until we decide to fully commit to nix.

Call-outs:

Because the omnibus code changes don't actually affect the PR, and only prevent codebuild from running in our CI, I will leave those to the next PR. But the change to the omnibus build would be:

    - identifier: s2nDynamicLoad
      buildspec: codebuild/spec/buildspec_ubuntu.yml
      env:
        compute-type: BUILD_GENERAL1_SMALL
        image: 024603541914.dkr.ecr.us-west-2.amazonaws.com/docker:ubuntu18codebuild
        privileged-mode: true
        variables:
          TESTS: dynamicload
          GCC_VERSION: '9'
          S2N_LIBCRYPTO: openssl-1.1.1

Testing:

Includes a test

Successful codebuild run: https://us-west-2.console.aws.amazon.com/codesuite/codebuild/024603541914/projects/s2nOmnibus/build/s2nOmnibus%3A20bf6549-10f1-4c5c-aa1d-8c220f0977f9?region=us-west-2

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[dougch]

Consider making the omnibus changes separate so we get CI runs from codebuild. Also still thinking about how to get this to pass under nix.

[toidiu]

Can we also add a test case scenario where we dont call cleanup and verify that this doesnt break

[bretambrose]

If you don't call cleanup this will blow up because the dangling destructor will get invoked after unload. If you want a solution where that doesn't happen then you'd need to completely rethink how thread locals and cleanup are managed.

[maddeleine]

The dlclose call will technically calls s2n_cleanup because of our atexit handler. So there wouldn't be a dangling destructor in this case.

https://linux.die.net/man/3/dlopen
Since glibc 2.2.3, atexit can be used to register an exit handler that is automatically called when a library is unloaded.

Although it is dependent on the glibc version I guess.

[toidiu]

Right I expect this to not fail if an application fails to all s2n_cleanup since the atexit callback will be invoked. Do we want to test that case here as well to confirm that