LN problem: certificate invalid

[bonomat]

I observed the following error on our e2e tests.

FAIL tests/ether_halight.ts (46.283s)
  ● han-ethereum-ether-halight-lightning-bitcoin-alice-redeems-bob-redeems

    timed out after 20000ms

      at Timeout.<anonymous> (node_modules/comit-sdk/src/util/timeout_promise.ts:18:14)

Log excerpt:

�[2mApr 16 00:45:41.460�[0m �[31mERROR�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=416e01ce-8eb8-4d60-9506-7ad73bda1211 role=Bob�[1m}�[0m: cnd::network::comit_ln: swap failed with error sending request for url (https://localhost:51225/v1/payments?include_incomplete=true): error trying to connect: The certificate was not trusted.

Caused by:
    0: error trying to connect: The certificate was not trusted.
    1: The certificate was not trusted.
�[2mApr 16 00:45:41.463�[0m �[32m INFO�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=416e01ce-8eb8-4d60-9506-7ad73bda1211 role=Bob�[1m}�[0m: cnd::network::comit_ln: swap finished

alice.log
cnd-alice.log
bob.log
cnd-bob.log
test_environment.log
parity.log
alice-lnd.log
bob-lnd.log

[bonomat]

Pinging @D4nte as the master of thunder and lightning :)

[D4nte]

@da-kami has the same issue locally and spent the day debugging it. I double checked myself and no problem. What version of Mac do you have @bonomat?

[D4nte]

I have Mojave and no problem, @da-kami seems to think he has the error since he upgraded his macbook.

We use reqwest and add the lnd tls cert to our requests (that's how it works).

The default feature flag of reqwest for TLS is default-tls. I asked @da-kami to pass native-tls or rustls-tls and see if it helps.

[da-kami]

I have Mojave and no problem, @da-kami seems to think he has the error since he upgraded his macbook.

We use reqwest and add the lnd tls cert to our requests (that's how it works).

The default feature flag of reqwest for TLS is default-tls. I asked @da-kami to pass native-tls or rustls-tls and see if it helps.

Correction:
We have it set to native-tls in the Cargo.toml.Am trying default-tlsand rustls-tls and see if it helps.

[da-kami]

Note:
I validated that the path to the certificate is correctly passed in (logged the path and the certificate).
lnd version is correct at lnd version 0.9.1-beta commit=v0.9.1-beta
(even reinstalled it after a go clean and system restart)

[da-kami]

When running with rustls-tls I get a different error:

�[2mApr 16 17:54:11.275�[0m �[32m INFO�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m: cnd::network::comit_ln: yielded event Started
�[2mApr 16 17:54:11.275�[0m �[32m INFO�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m: cnd::network::comit_ln: yielded event Opened
�[2mApr 16 17:54:11.275�[0m �[32m INFO�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m: cnd::swap_protocols::halight::connector: Certificate: Certificate
�[2mApr 16 17:54:11.291�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fa73aad9c163d6f9cac31016558326921d43303e2af1a8587bf1ccd91cbfc78da60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e980f1d106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a173c6723a5fd978121fc227d42f99dbb29257976d67ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a1736aeb4d81488d4574082303326595300c4875545bff�[1m}�[0m: cnd::btsieve::ethereum::web3_connector: Fetched block from web3: a4af6c34831629b0f8bb00cad14c8fcb17b0b381297a8d7669e86dba761f989f
�[2mApr 16 17:54:11.292�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fa73aad9c163d6f9cac31016558326921d43303e2af1a8587bf1ccd91cbfc78da60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e980f1d106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a173c6723a5fd978121fc227d42f99dbb29257976d67ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a1736aeb4d81488d4574082303326595300c4875545bff�[1m}�[0m:�[1mnew_block�[0m�[1m{�[0mblockhash=a4af6c34831629b0f8bb00cad14c8fcb17b0b381297a8d7669e86dba761f989f�[1m}�[0m: cnd::btsieve::ethereum: checking 0 transactions
�[2mApr 16 17:54:11.323�[0m �[33m WARN�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m: rustls::session: Sending fatal alert BadCertificate    
�[2mApr 16 17:54:11.323�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fa73aad9c163d6f9cac31016558326921d43303e2af1a8587bf1ccd91cbfc78da60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e980f1d106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a173c6723a5fd978121fc227d42f99dbb29257976d67ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a1736aeb4d81488d4574082303326595300c4875545bff�[1m}�[0m: cnd::btsieve::ethereum::web3_connector: Fetched block from web3: 2a609893ea4596bc0fafd37761c77fffbd491d7ee65344c231752e2b53549f4b
�[2mApr 16 17:54:11.323�[0m �[31mERROR�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=d9435252-3cb5-478c-8f17-22b4a6507c18 role=Bob�[1m}�[0m: cnd::network::comit_ln: swap failed with error sending request for url (https://localhost:57551/v1/payments?include_incomplete=true): error trying to connect: invalid certificate: CAUsedAsEndEntity

Caused by:
    0: error trying to connect: invalid certificate: CAUsedAsEndEntity
    1: invalid certificate: CAUsedAsEndEntity

when using default-tls it's the same error as for native-tls:

�[2mApr 16 18:03:00.267�[0m �[34mDEBUG�[0m cnd::http_api::routes: returning empty siren document because states are not yet completed
�[2mApr 16 18:03:00.267�[0m �[32m INFO�[0m http: - "GET /swaps/96f10b1c-98e8-4c4a-95df-c53b20041874 HTTP/1.1" 200 "-" "axios/0.19.2" 16.787817ms    
�[2mApr 16 18:03:00.272�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fcb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e981130106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a1735519c97b963eeb6b8aeb66a5fe9548a0e38371a8ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a173b3cbf6131b7a492fd1ea6747a6e12a4eb70bb52bff�[1m}�[0m: cnd::btsieve::ethereum::web3_connector: Fetched block from web3: 01d88eb66d8536326b18668c3354c867995d6c28a821de3ec7cb30b8ebafeb5b
�[2mApr 16 18:03:00.272�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fcb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e981130106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a1735519c97b963eeb6b8aeb66a5fe9548a0e38371a8ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a173b3cbf6131b7a492fd1ea6747a6e12a4eb70bb52bff�[1m}�[0m:�[1mnew_block�[0m�[1m{�[0mblockhash=01d88eb66d8536326b18668c3354c867995d6c28a821de3ec7cb30b8ebafeb5b�[1m}�[0m: cnd::btsieve::ethereum: checking 0 transactions
�[2mApr 16 18:03:00.320�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fcb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e981130106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a1735519c97b963eeb6b8aeb66a5fe9548a0e38371a8ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a173b3cbf6131b7a492fd1ea6747a6e12a4eb70bb52bff�[1m}�[0m: cnd::btsieve::ethereum::web3_connector: Fetched block from web3: 6208b7e7dee3d2fec125846f677b66da349330b0236fd5956fdb0c3a66aa79de
�[2mApr 16 18:03:00.329�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fcb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e981130106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a1735519c97b963eeb6b8aeb66a5fe9548a0e38371a8ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a173b3cbf6131b7a492fd1ea6747a6e12a4eb70bb52bff�[1m}�[0m: cnd::btsieve::ethereum::cache: Fetched block from connector: 6208b7e7dee3d2fec125846f677b66da349330b0236fd5956fdb0c3a66aa79de
�[2mApr 16 18:03:00.338�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fcb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e981130106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a1735519c97b963eeb6b8aeb66a5fe9548a0e38371a8ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a173b3cbf6131b7a492fd1ea6747a6e12a4eb70bb52bff�[1m}�[0m:�[1mnew_block�[0m�[1m{�[0mblockhash=6208b7e7dee3d2fec125846f677b66da349330b0236fd5956fdb0c3a66aa79de�[1m}�[0m: cnd::btsieve::ethereum: checking 0 transactions
�[2mApr 16 18:03:00.339�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fcb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e981130106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a1735519c97b963eeb6b8aeb66a5fe9548a0e38371a8ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a173b3cbf6131b7a492fd1ea6747a6e12a4eb70bb52bff�[1m}�[0m: cnd::btsieve::ethereum::web3_connector: Fetched block from web3: 01d88eb66d8536326b18668c3354c867995d6c28a821de3ec7cb30b8ebafeb5b
�[2mApr 16 18:03:00.339�[0m �[35mTRACE�[0m �[1malpha_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m:�[1mhtlc_deployed�[0m�[1m{�[0mexpected_bytecode=61012861000f6000396101286000f3361561007957602036141561004f57602060006000376020602160206000600060026048f17fcb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b60215114166100ae575b7f696e76616c69645365637265740000000000000000000000000000000000000060005260206000fd5b42635e981130106100eb577f746f6f4561726c7900000000000000000000000000000000000000000000000060005260206000fd5b7f72656465656d656400000000000000000000000000000000000000000000000060206000a1735519c97b963eeb6b8aeb66a5fe9548a0e38371a8ff5b7f726566756e64656400000000000000000000000000000000000000000000000060006000a173b3cbf6131b7a492fd1ea6747a6e12a4eb70bb52bff�[1m}�[0m:�[1mnew_block�[0m�[1m{�[0mblockhash=01d88eb66d8536326b18668c3354c867995d6c28a821de3ec7cb30b8ebafeb5b�[1m}�[0m: cnd::btsieve::ethereum: checking 0 transactions
�[2mApr 16 18:03:00.355�[0m �[31mERROR�[0m �[1mbeta_ledger�[0m�[1m{�[0mswap_id=96f10b1c-98e8-4c4a-95df-c53b20041874 role=Alice�[1m}�[0m: cnd::network::comit_ln: swap failed with error sending request for url (https://localhost:59627/v1/invoice/cb572e9b6ffa1c8078d50c907d52faeb1fae3a618c62f49ac0548c9a1e7d127b): error trying to connect: The certificate was not trusted.

Caused by:
    0: error trying to connect: The certificate was not trusted.
    1: The certificate was not trusted.

[bonomat]

@da-kami has the same issue locally and spent the day debugging it. I double checked myself and no problem. What version of Mac do you have @bonomat?

The error occurred on GitHub CI for the macos build.

[da-kami]

@da-kami has the same issue locally and spent the day debugging it. I double checked myself and no problem. What version of Mac do you have @bonomat?

The error occurred on GitHub CI for the macos build.

Yeah, an that is using Catalina: https://github.blog/changelog/2019-11-06-github-actions-macos-virtual-environment-updated-to-catalina/

[da-kami]

The error when using rustls-tls is interesting, but I am not sure how to go on:

https://briansmith.org/rustdoc/webpki/enum.Error.html#variant.CAUsedAsEndEntity

A CA certificate is veing used as an end-entity certificate.

There is an issue reported for that:
briansmith/webpki#114

and a fix on the way:
briansmith/webpki#127

(I checked and webpki is used by reqwest)

[da-kami]

Might be related: sfackler/rust-native-tls#151

[da-kami]

Found a workaround using native-tls:

seanmonstar/reqwest#182

[thomaseizinger]

Given the information here, the ideal fix is probably to ping the lnd guys and tell them to generate a valid certificate on startup :)

[da-kami]

Given the information here, the ideal fix is probably to ping the lnd guys and tell them to generate a valid certificate on startup :)

Created an issue: lightningnetwork/lnd#4201
Hope it is described in an understandable manner.