Python instance can not request oauth2 tokens

[jiangpengcheng]

context: https://streamnative.slack.com/archives/C03N9PFH5EX/p1666322103517279

@BewareMyPower thanks for looking into this

[jiangpengcheng]

I checked that the issue exists after version 2.9.2.22, and 2.9.2.21 is ok

[jiangpengcheng]

maybe caused by this one? https://github.com/streamnative/pulsar/commit/3211f91ae91239fb58bc1c709b66f41dcc9023ed

[BewareMyPower]

Could you give a OAuth config to reproduce it? (If it contains some private info, you can send the configs to me privately in Slack)

[BewareMyPower]

It can be simply reproduce inside the streamnative/pulsar:2.9.2.23 container by running the following Python script:

#!/usr/bin/env python3
from pulsar import AuthenticationOauth2

auth = AuthenticationOauth2(auth_params_string='''{
    "type": "client_credentials",
    "issuer_url": "https://dev-kt-aa9ne.us.auth0.com",
    "client_id": "xxx",
    "client_secret": "xxx"
}''')

Output:

2022-11-02 05:29:40.613 ERROR [140071187773248] AuthOauth2:223 | Response failed for getting the well-known configuration https://dev-kt-aa9ne.us.auth0.com. Error Code 77: error setting certificate verify locations:
  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

The issuer url can by any address.

It can be also reproduced by a C++ application:

// test2.cc
#include <pulsar/Client.h>

int main(int argc, char* argv[]) {
    std::string params = R"({
    "type": "client_credentials",
    "issuer_url": "https://dev-kt-aa9ne.us.auth0.com",
    "client_id": "xxx",
    "client_secret": "xxx"
})";
    auto auth = pulsar::AuthOauth2::create(params);
    auth.reset();
    return 0;
}

$ cp /usr/local/lib/python3.8/dist-packages/_pulsar.cpython-38-x86_64-linux-gnu.so _pulsar.so
$ cp /usr/local/lib/python3.8/dist-packages/_pulsar.cpython-38-x86_64-linux-gnu.so libpulsar.so
# Copy the C++ headers under ./pulsar/ directory
$ g++ test2.cc -I . \
 -L . -Wl,-rpath=. -lpulsar \
 -L /usr/lib/x86_64-linux-gnu -lpython3.8 \
 -D_GLIBCXX_USE_CXX11_ABI=0
$ ./a.out
2022-11-02 06:29:59.303 ERROR [140049929804224] AuthOauth2:223 | Response failed for getting the well-known configuration https://dev-kt-aa9ne.us.auth0.com. Error Code 77: error setting certificate verify locations:
  CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

I will confirm the Python client build process when building the streamnative/pulsar image and see if there is any error.

[BewareMyPower]

The wheel under /pulsar/pulsar-client is broken. I tried installing it in my Ubuntu 20.04 system and can reproduce it.

[BewareMyPower]

It's fixed by reverting apache/pulsar#16064. See apache/pulsar#16064 (comment).

It's already reverted in all active branches in Pulsar.

[BewareMyPower]

Reopen this issue since we need to restore apache/pulsar#16064

[BewareMyPower]

According to https://stackoverflow.com/questions/3160909/how-do-i-deal-with-certificates-using-curl-while-trying-to-access-an-https-url, the root cause is the bundled certificates in RedHat based Linux is different from the path in Debina based Linux. We can fix it by running the following commands before tests:

# Install the certificates to /etc/ssl/certs/ca-certificates.crt
apt-get install -y ca-certificates
# Copy it to the bundled path
mkdir -p /etc/pki/tls/certs
cp /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt

@yaalsn Should we change the CI verification workflow?