Example in the README will not run because of "pods is forbidden"

[judepereira]

I'm running the example in the README for the repo, and I cannot get it to work:

HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list pods at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403}

If the client is supposed to read the default kube config (which I think it does), it somehow cannot authenticate correctly. Is this a bug? Or is my usage incorrect?

[tomplus]

The example uses a method list_pod_for_all_namespaces which is equivalent of command:

kubect get pods --all-namespaces

Try to check if kubectl can list all pods. It looks like your user system:anonymous doesn't have enough permissions. You can try to use method list_namespaced_pod("your-namespace") or grant permissions to the user (RBAC).

[judepereira]

Yes, I can run kubectl get pods --all-namespaces in the terminal correctly.

Why is it using system:anonymous? Shouldn't it use whatever credentials are there via ~/.kube/config?

[tomplus]

It seems like the bug. Could you provide some details from ~/.kube/config? of course with obfuscated certs, addresses etc.

[judepereira]

Yes, here are the contents, after removing all the extra clusters:

apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://localhost:6443
  name: docker-for-desktop-cluster
contexts:
- context:
    cluster: docker-for-desktop-cluster
    namespace: asyncy-system
    user: docker-for-desktop
  name: docker-for-desktop
current-context: docker-for-desktop
kind: Config
preferences: {}
users:
- name: docker-for-desktop
  user:
    client-certificate-data: ...
    client-key-data: ...

[tomplus]

Looks good. Is it possible that one of the extra clusters from you kubeconfig has the same name? There was such issue in the official library and probably it's not ported yet kubernetes-client/python#445 Could you check if the official synchronous client has the same problem? Thanks.

[judepereira]

I ran this example in a clean venv, and it worked. I also checked if I had any duplicates, and I do not have any.

[judepereira]

Any way I can turn on extreme debugging in your client?

[tomplus]

I'm afraid there are not debug logs. I've compared these libraries and I can't find any significant differences...

Can I try to recreate your environment? Is docker-for-desktop a K8s cluster started by docker on Windows/MacOs?

[judepereira]

Yes, it's a k8s cluster started by docker on macOS 10.13.6.

[tomplus]

Unfortunately I don't have access to macOS. I added debug logs (#45) which may help. Could you check this code?

import asyncio
import logging

from kubernetes_asyncio import client, config

async def main():
    await config.load_kube_config()
    
    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = await v1.list_pod_for_all_namespaces()
    
    for i in ret.items:
        print(i.status.pod_ip, i.metadata.namespace, i.metadata.name)


if __name__ == '__main__':
    logging.basicConfig(level=logging.DEBUG)
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()

[judepereira]

@tomplus Here's what the output is:

DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:root:kubeconfig loader - current-context docker-for-desktop, cluster docker-for-desktop-cluster, user docker-for-desktop, provider None
DEBUG:root:Try to load user token
DEBUG:root:Try to use username and password
Listing pods with their IPs:
DEBUG:kubernetes_asyncio.client.rest:response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list pods at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403}

Traceback (most recent call last):
  File "test.py", line 20, in <module>
    loop.run_until_complete(main())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "test.py", line 11, in main
    ret = await v1.list_pod_for_all_namespaces()
  File "/Users/jude/test/lib/python3.6/site-packages/kubernetes_asyncio-1.0.0b8-py3.6.egg/kubernetes_asyncio/client/api_client.py", line 153, in __call_api
  File "/Users/jude/test/lib/python3.6/site-packages/kubernetes_asyncio-1.0.0b8-py3.6.egg/kubernetes_asyncio/client/rest.py", line 192, in GET
  File "/Users/jude/test/lib/python3.6/site-packages/kubernetes_asyncio-1.0.0b8-py3.6.egg/kubernetes_asyncio/client/rest.py", line 182, in request
kubernetes_asyncio.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Sun, 07 Oct 2018 16:29:38 GMT', 'Content-Length': '222')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list pods at the cluster scope","reason":"Forbidden","details":{"kind":"pods"},"code":403}


ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x1096092e8>
ERROR:asyncio:Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x1094c47c8>, 18074.071092205)]']
connector: <aiohttp.connector.TCPConnector object at 0x109609cc0>

[tomplus]

I've check code again and it looks like the problem is caused by this settings insecure-skip-tls-verify: true. Could you try to remove this flag from your kubeconfig?

The verify_ssl, ssl_context, fingerprint and ssl parameters are mutually exclusive in aiohttp, so this library won't pass certs if you unset verify_ssl... I can change the behavior after your confirmation. If certs are provider this insecure-skip-tls-verify will be ignored.

[judepereira]

@tomplus We're making some progress here - after removing insecure-skip-tls-verify, I can see that the client is trying to use the cert, but fails because:

aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host localhost:6443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841)]

Perhaps the interpretation of that flag is incorrect in the library? insecure-skip-tls-verify should be respected, however, when verifying the validity of the TLS certificate, it's okay to trust it (as the root CA for that will not be installed in the system - the root CA in this case will be a self signed Kubernetes cert).

[tomplus]

Fixed! Now this flag impacts on server verification only. Thanks for your patience :)

[judepereira]

Thank you so much @tomplus! I can verify that it's working well now.