Issues verifying certificate connecting with HTTPS

[theodorton]

Crystal version

~/code/nabobil/autotoll · (master±)
⟩ crystal -v
Crystal 0.23.1 (2017-10-12) LLVM 4.0.1

How to reproduce

# test.cr
require "http/client"

client = HTTP::Client.new("maps.googleapis.com", tls: true)
client.get "/"

Then run crystal run test.cr.

Expected outcome

No errors raised, or some redirect.

Actual outcome

SSL_connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (OpenSSL::SSL::Error)
0x10d990e45: *CallStack::unwind:Array(Pointer(Void)) at ??
0x10d990de1: *CallStack#initialize:Array(Pointer(Void)) at ??
0x10d990db8: *CallStack::new:CallStack at ??
0x10d98d615: *raise<OpenSSL::SSL::Error>:NoReturn at ??
0x10da33571: *OpenSSL::SSL::Socket::Client#initialize<TCPSocket, OpenSSL::SSL::Context::Client, Bool, String>:Nil at ??
0x10da333ec: *OpenSSL::SSL::Socket::Client::new:context:sync_close:hostname<TCPSocket, OpenSSL::SSL::Context::Client, Bool, String>:OpenSSL::SSL::Socket::Client at ??
0x10da29c91: *HTTP::Client#socket:(OpenSSL::SSL::Socket+ | TCPSocket+) at ??
0x10da299eb: *HTTP::Client#exec_internal_single<HTTP::Request>:(HTTP::Client::Response | Nil) at ??
0x10da26f7c: *HTTP::Client#exec_internal<HTTP::Request>:HTTP::Client::Response at ??
0x10da26e72: *HTTP::Client#exec<HTTP::Request>:HTTP::Client::Response at ??
0x10da26bdd: *HTTP::Client#exec<String, String, Nil, Nil>:HTTP::Client::Response at ??
0x10da26bb7: *HTTP::Client#get<String>:HTTP::Client::Response at ??
0x10d97bcba: __crystal_main at ??
0x10d98c678: main at ??

Notes

• This also happens when running the code within the official docker image for 0.23.1.
• I've managed to get it to pass randomly, but I believe there is some cipher requirement and that the certificates differ somewhat between the servers behind this domain
• When running openssl s_client -connect maps.googleapis.com:443 -tls1_2 I get:

CONNECTED(00000003)
depth=2 C = US, O = GeoTrust Inc., CN = GeoTrust Global CA
verify return:1
depth=1 C = US, O = Google Inc, CN = Google Internet Authority G2
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google Inc, CN = *.googleapis.com
verify return:1
---
...
...

• It works if I setup a similar example in Ruby:

require 'net/http'
require 'uri'

uri = URI('https://maps.googleapis.com/')

Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
  request = Net::HTTP::Get.new uri
  response = http.request request # Net::HTTPResponse object
  puts response
end

• In other words, I don't believe this is an issue with my openssl configuration.

[Sija]

Duplicate of #3477.

See also composer/composer#3346 and relevant question on SO.

[theodorton]

I'm not sure, I'm able to connect with TLS to github.com:

HTTP::Client.get("https://github.com/") # This works
HTTP::Client.get("https://maps.googleapis.com/") # This doesn't

I belive there might be something with the allowed cipher suites as SSLLabs Test results for maps.googleapis.com show that the root certificate is insecure.

[asterite]

I face this problem with some URLs. I think it depends on the version of openssl. Ruby ships with one version of openssl and that's very good: reproducible builds and behavior no matter what your system is. In Crystal we dynamically link to the openssl that's present in the system and it leads to these random and hard to reproduce problems (because we have to have your version of openssl in our machines).

I wish we could just ship Crystal with one version of openssl, one version of llvm, one version of libpcre, one version of libxml2, etc., so that everything works exactly the same everywhere...

[RX14]

@asterite And then what if openssl has a security bug? If we distribute our own libraries we are now distro maintainers. We have to follow the security bugs closely with a quick response, and possibly backport patches, and deal with a whole bunch of things we don'y have time to do. This is the distro maintainer's job, not ours, it's already done for us.

I've never had an openssl problem with crystal on archlinux, i've only seem people have problems on osx and on debian-based (read: ancient versions of openssl) distros.

[theodorton]

I don't have any strong opinions on whether to bundle openssl with the compiler or not, but I think this doing VERIFY_NONE as a workaround (which is mentioned in the linked issues) here unsatisfactory.

I tried to use OpenSSL::SSL::Context::Client#add_x509_verify_flags(OpenSSL::SSL::X509VerifyFlags:: TRUSTED_FIRST) which was mentioned as a fix in a Stack Overflow issue I found yesterday. I'm unable to find it again now, but the problem I ran into was that OpenSSL::SSL::X509VerifyFlags was not defined. Seems like there's a compiler flag for that constant that wasn't set.

It seems highly related, but I'm not sure? I'm building Crystal via Homebrew. Could it be that it's missing some SSL/crypto lib while compiling? Or that the compiled bottle comes without this support?

Edit: I'm trying to install from homebrew with --build-from-source to see if that resolves the issue.
Update: No, it doesn't help. I'm still getting undefined constant OpenSSL::OPENSSL_102 errors when attempting to use X509VerifyFlags.

[RX14]

It appears to me that openssl's API is a mess, perhaps we should implement the libtls API using openssl (in C, how openssl wants us to), and then use that as the base for crystal's TLS module and remove OpenSSL.

[theodorton]

The workaround here is:

context = OpenSSL::SSL::Context::Client.insecure
client = HTTP::Client.new("maps.googleapis.com", tls: context)

I'm not sure if that's a very good workaround though, as if I point the client to expired.badssl.com, I'm not getting any errors.

[ysbaddaden]

Hence the insecure: you basically accept whatever OpenSSL supports —I'm not even sure it verifies certificates. Would you be willing to take the Client.insecure and tweak it until you find the culprit in defaults?

[theodorton]

@ysbaddaden Yes, we're ok with Client.insecure for now, but it doesn't seem to verify certificates. I think that should probably be noted in the documentation. Currently the docs say:

Use this only if undoing the defaults that new sets is too much hassle.

Not sure if I can spend more time at the moment looking into the defaults. I'm not very fluent in TLS/SSL terminology so I'd just have to guess what the different options represent. The failures were pretty consistent with that hostname, even when I ran it in the Docker image for 0.23.1.

I still think this is highly relevant:

I'm still getting undefined constant OpenSSL::OPENSSL_102 errors when attempting to use X509VerifyFlags.

As I believe using the OpenSSL::SSL::X509VerifyFlags:: TRUSTED_FIRST might work.

[vladfaust]

Any updates? I'm basically unable to connect to any of Google's APIs 😕

require "http/client"
client = HTTP::Client.new(URI.parse("https://google.com"))
p client.get("/")

It raises the same certificate verify failed (OpenSSL::SSL::Error).

[will]

Just chiming into say that I'm having this problem on heroku with google oauth.

Also since the oauth::client doesn't take a tls: context, the workaround of not verifying doesn't work, unless you monkey patch

[waj]

I just sent a PR with a potential fix. Please, let me know if it works for you and also provide some comments on the solution if possible.

[codenoid]

Crystal 0.24.1 (2017-12-22)

LLVM: 4.0.0
Default target: x86_64-unknown-linux-gnu

still error when accessing maps.google.com

SSL_connect: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed (OpenSSL::SSL::Error)
  from /usr/share/crystal/src/openssl/ssl/socket.cr:34:9 in 'initialize'
  from OpenSSL::SSL::Socket::Client::new:context:sync_close:hostname<TCPSocket, OpenSSL::SSL::Context::Client, Bool, String>:OpenSSL::SSL::Socket::Client
  from /usr/share/crystal/src/http/client.cr:657:5 in 'socket'
  from /usr/share/crystal/src/http/client.cr:500:5 in 'exec_internal_single'
  from /usr/share/crystal/src/http/client.cr:486:5 in 'exec_internal'
  from /usr/share/crystal/src/http/client.cr:482:5 in 'exec'
  from /usr/share/crystal/src/http/client.cr:585:5 in 'exec'
  from /usr/share/crystal/src/http/client.cr:612:7 in 'exec'
  from /usr/share/crystal/src/http/client.cr:329:3 in 'get'
  from src/facebook.cr:32:5 in '__crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:11:3 in '_crystal_main'
  from /usr/share/crystal/src/crystal/main.cr:112:5 in 'main_user_code'
  from /usr/share/crystal/src/crystal/main.cr:101:7 in 'main'
  from /usr/share/crystal/src/crystal/main.cr:135:3 in 'main'
  from __libc_start_main
  from _start
  from ???

oh wait, i'll try @waj PR

Update fixed with @waj PR (#5601)

Update

Python Script :

import requests

headers = {
    'Host': 'maps.googleapis.com',
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://google-developers.appspot.com/maps/documentation/utils/geocoder/embed',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Cache-Control': 'max-age=0',
}

params = (
    ('', ''),
    ('address', 'Kalibata City'),
)

response = requests.get('https://maps.googleapis.com/maps/api/geocode/json', headers=headers, params=params)

print response.text

""" result (200) :
{
   "results" : [
      {
         "address_components" : [
            {
               "long_name" : "Kalibata City",
               "short_name" : "Kalibata City",
               "types" : [ "premise" ]
            },
            {
               "long_name" : "1",
               "short_name" : "1",
               "types" : [ "street_number" ]
            },
            {
               "long_name" : "Jalan Raya Kalibata",
               "short_name" : "Jl. Raya Kalibata",
               "types" : [ "route" ]
            },
            {
               "long_name" : "Rawajati",
               "short_name" : "Rawajati",
               "types" : [ "administrative_area_level_4", "political" ]
            },
            {
               "long_name" : "Pancoran",
               "short_name" : "Pancoran",
               "types" : [ "administrative_area_level_3", "political" ]
            },
            {
               "long_name" : "Kota Jakarta Selatan",
               "short_name" : "Kota Jakarta Selatan",
               "types" : [ "administrative_area_level_2", "political" ]
            },
            {
               "long_name" : "Daerah Khusus Ibukota Jakarta",
               "short_name" : "Daerah Khusus Ibukota Jakarta",
               "types" : [ "administrative_area_level_1", "political" ]
            },
            {
               "long_name" : "Indonesia",
               "short_name" : "ID",
               "types" : [ "country", "political" ]
            },
            {
               "long_name" : "12750",
               "short_name" : "12750",
               "types" : [ "postal_code" ]
            }
         ],
         "formatted_address" : "Kalibata City, Jl. Raya Kalibata No.1, Rawajati, Pancoran, Kota Jakarta Selatan, Daerah Khusus Ibukota Jakarta 12750, Indonesia",
         "geometry" : {
            "bounds" : {
               "northeast" : {
                  "lat" : -6.2557049,
                  "lng" : 106.8546168
               },
               "southwest" : {
                  "lat" : -6.258145,
                  "lng" : 106.8493882
               }
            },
            "location" : {
               "lat" : -6.2572684,
               "lng" : 106.8521429
            },
            "location_type" : "GEOMETRIC_CENTER",
            "viewport" : {
               "northeast" : {
                  "lat" : -6.255575969708497,
                  "lng" : 106.8546168
               },
               "southwest" : {
                  "lat" : -6.258273930291502,
                  "lng" : 106.8493882
               }
            }
         },
         "place_id" : "ChIJJ1jytbLzaS4RcqOnV26PRPs",
         "types" : [ "premise" ]
      }
   ],
   "status" : "OK"
}
"""

Crystal :

require "http/client"
a = HTTP::Client.get("https://maps.googleapis.com/maps/api/geocode/json?&address=Kalibata%20City", headers: HTTP::Headers{"User-Agent" => "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0", "Accept" => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language" => "en-US,en;q=0.5", "Accept-Encoding" => "gzip, deflate, br", "Referer" => "https://google-developers.appspot.com/maps/documentation/utils/geocoder/embed", "Connection" => "keep-alive", "Upgrade-Insecure-Requests" => "1", "Cache-Control" => "max-age=0"})
puts a.body

# result (200) :
# �VM@�+:Y�^�QwGL�����zۥ�;O�&�zU!dDp}E?t-�E��F1#�S�at�PHq�V@@D�1DoR6�N�.j)z76���+]Ig��(a<n32L�X.;x�t(Mh"g)[�@! xI`*TD$!+y/,���-yϙ<�3xL^�0*gV~[s8��MWKjwcr4kb*:Q�m."ߒn8�9*{SQP���3TVR7�E�h_E
#    Du5{ZGԡE;i,YFk4j_<hT	U)8>j�ixRm1y`<��X&xJDC˱<s49+ii8#o�nimD}S6�B�$Q9q<X}
#                                                                            �YПv.~ @G3X̕a�}k 

• lolno

[codenoid]

@waj @ysbaddaden @RX14 the endpoint has not been verified ?

UPDATE : when maps.googleapis.com returning 400 status code, the response.body is not encrypted (just like usually response body / html page that showing a error)

Replace %20 with a space :

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (Bad Request)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>That’s an error.</ins>
  <p>Your client has issued a malformed or illegal request.  <ins>That’s all we know.</ins>

[RX14]

@codenoid it's because you specify Accept-Encoding in the request headers. You're telling crystal you accept the gzip encoding, and so crystal gives you back your gzip encoding.

Please don't copy your browser's headers into crystal code, just

require "http"

HTTP::Client.get("https://maps.googleapis.com/maps/api/geocode/json?&address=Kalibata%20City")

is enough and you get a nice response by default.

[codenoid]

wat, why it's work ? because when i try with simple GET, i got 400 :/ aarrrg

UPDATE

at 02:30 and i;m tired, i don't encode the url

UPDATE

at 08:59, i try @RX14 code, and it's give me a wanted response, so fixed by sleep 👍

UPDATE

I sleep at 05:00

UPDATE

I react :hooray: to @waj comment

[sam0x17]

I am also experiencing this issue here in the context of a docker image running in lambda via up.sh: Sija/raven.cr#47 (comment)

[benbonnet]

Hello all;

quite new to crystal; focusing on getting the smallest docker image (so, statically building a crystal app).

Currently stuck on the issue here; I was thinking about using an intermediate to retrieve the required certs :

FROM crystallang/crystal:latest as compiler
RUN mkdir /app
COPY . /app
RUN mkdir /app/bin
WORKDIR /app
RUN shards --production
RUN crystal build /app/src/appname.cr --static --release --no-debug -o /app/bin/appname

FROM alpine as certs-builder
RUN apk update && apk upgrade && apk add --no-cache ca-certificates
RUN update-ca-certificates

FROM busybox:glibc
COPY --from=compiler /app/bin/appname .
COPY --from=certs-builder /etc/ssl/certs /etc/ssl/certs

CMD ["./appname"]

Still failing.

Would there be a way to achieve something similar by specifying the path to the certs in the tls context provided to ? Or should I choose another approach ?

It feels awkward to use OpenSSL::SSL::Context::Client.insecure, although it make it all work as expected

[Blacksmoke16]

@benbonnet The problem is statically linking is only supported in Alpine linux. See https://crystal-lang.org/reference/using_the_compiler/index.html#creating-a-statically-linked-executable and https://crystal-lang.org/2020/02/02/alpine-based-docker-images.html.

[benbonnet]

@Blacksmoke16
As shown above it is first built within crystallang/crystal:latest.
On the final step; the app bin was copied through a FROM scratch in the last step.
Had to move on and use FROM busybox:glibc (otherwise it could not resolve hostnames; among other problems).
As this slight change enabled a missing capability; am in hope to find a way to just get what's needed be able to verify the certs

fully statically linked are only possible within alpine; but in case the only requirements was certs verification, is there a way to only get it to have a working final image ?

[Blacksmoke16]

@benbonnet I guess what I'm getting at is couldn't you statically link the binary in the Crystal alpine image, then move it to another smaller Alpine image and just use that? Like I suggested in #6099 (comment)? That's what I do and it's been working fine for me.

[benbonnet]

@Blacksmoke16 well you might get a way larger image (10x maybe ?). I do know my asking is completely overkill and we would still be fine by not even building statically.
But I've been super hyped to see how the smallest image possible would reduce our pipeline deployment and functions startup (we're using cloudrun, such a small image cold starts in 200ms or less)

thanks a lot for your link & help; i might follow that way

[Blacksmoke16]

@benbonnet

docker images
REPOSITORY       TAG                         IMAGE ID            CREATED             SIZE
busybox          glibc                       b0748d3617e4        9 days ago          5.21MB
alpine           3.12                        389fef711851        3 weeks ago         5.57MB

Size wise they're essentially the same, but with alpine you have a package manager and such. Granted this is not including the size of your Crystal binary, or other dependencies you may need.

EDIT: Might also be able to get away with not statically linking if it's also going to run in Alpine. However I'm not sure what the size comparison is to a static binary versus the libs themselves.

[straight-shoota]

Still failing.

What exactly is failing? Is it related to this issue or a different one? From the static linking discussion it feels like something else, which would probably fit better to https://forum.crystal-lang.org/

Not sure if dynamic linking is a factor for your issue, but when you copy a dynamically linked binary to a different system, you need to make sure it has matching dynamic libraries available.
I would strongly recommend to link fully statically (can use the alpine image for that). That causes less friction with moving between systems and should help you goal to reach minimal binary size.