[Bug]: HAProxy ca-file manipulation broken since 17c9e92b #8222
Comments
|
Here the output of the reg-test with --enable-debug on wolfssl. |
|
Hello @wlallemand I am trying to reproduce using your provided steps but am running into a problem. I tried to google around for a solution but did not find anything, I was wondering if you knew how to resolve these errors so I can reproduce? |
|
Hello @ColtonWilley, Please compile the VTest tool there https://github.com/wlallemand/VTest/tree/haproxy-sd_notify and set VTEST_PROGRAM when launching the reg-test.
I'm trying to isolate the code which does not work correctly so we can have a more precise idea of what's going on. |
|
Don't bother with haproxy, here is a reproducer. Compile it this way:
Use a PEM file in parameter, it should return the number of unique X509 entries in the PEM. With OpenSSL the number is right, but with WolfSSL 5.7.4 it's always 0. |
|
First off thank you so much for creating this, as somebody not familiar with haproxy it makes everything much easier on my end. Unfortunately though I was still not able to reproduce, it yielded identical results to openssl for me. Do you have a specific certificate example that is failing for you? |
|
This PEM file from our CI is giving me the following results: $ ./load_ca_openssl common.pem.txt $ ./load_ca_wolfssl common.pem.txt |
|
@wlallemand Thank you for the example file, I am now successfully able to reproduce. I will update you as soon as I have identified the root cause for this discrepancy. |
|
@wlallemand I have a PR up here #8226 that should resolve this issue. Please let me know if this solves the haproxy test failure, or if the issue persists. |
|
Thanks for the PR, that fixed part of the problem but I'm now having a SEGV in a related part: The problem is in this function which is a simple equivalent to what https://github.com/haproxy/haproxy/blob/master/src/ssl_ckch.c#L1751 I can try to isolate the code if you need! |
|
Here is the new reproducer, my guess is that X509_STORE_add_cert() is not incrementing the refcount so sk_X509_INFO_pop_free() frees everything and accessing the cert with cert_get_pkey_algo() can't work since the data are not here anymore. |
|
@wlallemand Again I cant thank you enough for your efforts in isolating the problem. I have already reproduced this segfault. I will let you know when I have a PR with a fix available to test. |
|
@wlallemand There is definitely an issue with the internal refcount handling, unfortunately managing this flow in addition to the hundreds of other use cases is proving a bit trickier than initially anticipated. I am on holiday tomorrow but will continue to work this problem when I return on Friday. |
No problem, I understand the pain of debugging in a whole new environment :-)
I guess it's more difficult than anticipated if the ASAN tests were right with the current implementation! Thanks for the quick feedback! |
|
@wlallemand I have a new PR up here #8233 that should hopefully resolve your issues. Please let me know how it goes. |
|
@ColtonWilley I just tested it and the refcount issue seem to be fixed, thanks a lot! But unfortunately I still have other issues, the reg-tests are still failing, I suspect something like X509_get_serialNumber() but I have to take some time to investigate more. |
|
@wlallemand Do you have any specific failures you need me to take a look at? If so please let me know. |
|
@ColtonWilley Sorry I didn't isolate the problem yet, I'll update you soon! Thanks |
|
Hi @ColtonWilley, sorry for the late reply, I was busy on other projects. It's quite difficult to reproduce the exact problem outside HAProxy. I tried again today with wolfssl master (commit 00f83fa) I don't really get what's going on, that could still be a mess with the refcounts. I compiled HAProxy with -fsanitize and run the following regtest: What I can see is some reuse-after-free: The trace is weird, wolfSSL_X509_STORE_get0_objects() seems to free the objects for an unknown reason, I didn't get the exact trace on my reproducer program but I still got something: However, when commenting a function (get_certificate_count()) which does a X509_STORE_get0_objects() previously, I could make it work: |
|
Hi @wlallemand , Colton is out on holiday until after new year. I am trying to get another engineer assigned to continue this investigation. Thanks, |
|
Hi @dgarske, Thanks! I will also be in holidays this week and the one after. I hope I gave enough details to investigate the problem. |
Contact Details
[email protected]
Version
v5.7.4
Description
Hello,
After updating our CI with the latest WolfSSL version, I found out that 2 of our regression tests were broken. v5.7.2 is behaving correctly, but v5.7.4 is broken.
After bisecting this is the first bad commit that introduced the problem:
The behavior of the X509_STORE does not seem to match the one of OpenSSL anymore, we don't have specific Wolfssl code in this part, we are only using the OpenSSL API.
I didn't try to reproduce the problem outside HAProxy yet, but I could try if that helps.
Reproduction steps
After cloning the master of haproxy and wolfssl in /tmp:
git bisect run sh testhaproxy.sh
testhaproxy.sh:
#!/bin/sh
./autogen.sh
./configure --enable-quic --enable-haproxy --enable-debug
make
cd /tmp/haproxy/
make -j8 TARGET=linux-glibc USE_OPENSSL_WOLFSSL=1 SSL_LIB=/tmp/wolfssl/src/.libs/ SSL_INC=/tmp/wolfssl/ ADDLIB=-Wl,-rpath,/tmp/wolfssl/src/.libs/
make reg-tests -- --debug reg-tests/ssl/new_del_ssl_cafile.vtc
Relevant log output
No response
The text was updated successfully, but these errors were encountered: