0.5.5 SSL issues

[pjenvey]

The features=[ "openssl"] causes SSL issues, seen on stage logs:

2020-08-06T19:42:59.394543836Z Corruption detected. E 
2020-08-06T19:42:59.394603261Z error:14187180:SSL routines:ssl_do_config:bad value E 
2020-08-06T19:42:59.394609172Z     Decryption error: TSI_DATA_CORRUPTED E 

One might assume the target_vendor="ubuntu" wouldn't affect the debian:buster-slim docker env we use, but there's no other reason this would begin happening on 0.5.5.

[jrconlin]

Prior to this version, the app would fail during init because of conflicts in openssl and the grpcio included boringssl.

It's targeted to ubuntu because I was unaware of it impacting anything else, but there appears to be a different issue at play.

Does it make sense to just set feature="openssl" universally?

[jrconlin]

Huh, are we terminating TLS on our machines or on an ELB equivalent? I'm guessing we're doing it on each server and we run a round-robin load balancer?

[pjenvey]

Does it make sense to just set feature="openssl" universally?

I doubt that would affect the docker build since I'm guessing the ubuntu target is being triggered. Did we confirm it's being enabled on there from cargo tree -e features?

Huh, are we terminating TLS on our machines or on an ELB equivalent? I'm guessing we're doing it on each server and we run a round-robin load balancer?

We don't terminate TLS, the LB does. The GRPC connections to Spanner are over TLS though.

[pjenvey]

JR confirmed the grpcio openssl feature is not being picked up on the Docker build, so it's not the cause.

(also rustc --print=cfg shows what the current env's vendor is).

All we know is this started happening in 0.5.5 on stage. It first occurred an hour after it was deployed. There's very little difference between 0.5.4 and 0.5.5

0.5.4...0.5.5

0.5.4 was on stage for a couple days before 0.5.5 (there wasn't a lot of traffic, but neither was there on 0.5.5). I can't easily tell if the libssl-dev install on the docker changed in between the 2 releases (the docker step for it was "skipped" with no log output because it hit the docker layer caching) but it shouldn't matter because grpcio's using boringssl underneath anyway.

[jrconlin]

I'm also not able to replicate the problem locally using a docker build. I built an image using the Dockerfile and was able to use grpcio to connect to spanner remotely without the above error. I don't think this may be a problem with the docker image building built on circleci, but at this point, I don't have any ideas.

[pjenvey]

See #774 (comment) for the likely culprit

[pjenvey]

Confirmed this went away on stage w/ #774's fix (in 0.5.6)