Improved message when not authorised to list namespaces

[vgibilmanno]

I can't use kubebox when I only have access to a namespace in a cluster. It would be awesome if I could use kubebox just for the namespace I have access to, ignoring everything outside the namespace.

[astefanutti]

You should be able to use Kubebox just for the namespaces you're granted access to. What error do you face?

[vgibilmanno]

The login fails with "Authentication failed" and I'm being asked for an username and a password. The cluster runs in AWS and I have the AWS credentials set. Because of that I don't really have a username or password to set. Kubectl can be executed perfectly fine.

[astefanutti]

AWS EKS is supposed to be working :(

There have been some issues that may provide some context: #32 #60.

What version of Kubebox do you use? Could you provide the content of your kubeconfig file?

In the meantime, you can run:

$ aws eks get-token --cluster-name [my-cluster] 

To get a token to authenticate with Kubebox.

[vgibilmanno]

Of course :) that's the content of the kubeconfig. I hope it's ok, that I replaced personal information with placeholders. But I didn't change the structure of the content.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://OMITTED.amazonaws.com
  name: SOMENAME
contexts:
- context:
    cluster: SOMENAME
    user: SOMENAME
  name: SOMENAME
current-context: SOMENAME
kind: Config
preferences: {}
users:
- name: SOMENAME
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - SOMEOTHERNAME
      - -r
      - arn:aws:iam::SOMEROLEID:role/SOMEROLE
      command: aws-iam-authenticator
      env: null

I'm using kubebox v0.6.1

How can I replace the token in the login screen? I'm too stupid right now sry

[astefanutti]

Thanks for the details. Nothing obvious so we'll have to try reproducing.

There is a token field in the login window, below the username and password fields. You can paste the token retrieve from AWS CLI.

[johnpoth]

@vgibilmanno perhaps some useful info would be to run the command locally yourself:

aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE and maybe show us the output ? Of course censuring private data. Thanks !

[vgibilmanno]

@johnpoth I get the following

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1alpha1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2019-11-27T15:51:10Z",
        "token": "SOMETOKEN"
    }
}

[astefanutti]

@johnpoth could it be that the provideAuth method does not get a first token with refresh_token when there is no token information?

[astefanutti]

@johnpoth the refresh_token method returns false when there is no expiration date:

console.log((undefined - Date.now()) < 5000)
> false

So we may have to add a condition to check whether a token exists:

kubebox/lib/auth/exec.js

Line 14 in f8cb4db

if (this.has_token_expired()) {

WDYT?

[johnpoth]

Hi @astefanutti so the expired is always set to now:

ExecAuthProvider ... this.expiry = Date.now();

So the token is always refreshed even if it's still valid on the first run. Therefore I think the problem lies elsewhere but I may be wrong...

@vgibilmanno have you tried pasting the TOKEN you got from running the command into the kubebox login screen ? This should help narrow things down a bit... thanks

[astefanutti]

@johnpoth I've just realised the expiry is set there:

kubebox/lib/config/user.js

Line 143 in f8cb4db

this.expiry = Date.now();

It seems it'd be easier to reason about if it would not be initialised... Could it be the login sequence takes less than 5ms so that the check fails?

[vgibilmanno]

@johnpoth I can't paste the token in the login screen. The paste command is not working and I would have to type in the whole token manually. I'm using alpine linux v3.10 in case that's relevant.

[johnpoth]

@astefanutti I hadn't thought about that... judging from the feedback #60 I'm thinking it might be something different ?
That being said I don't see what could be causing this from the Kubebox Exec plugin perspective...

It may be interesting for @vgibilmanno to use the Token directly and see if Kubebox can connect to the cluster ?

[astefanutti]

@vgibilmanno could you update your kubeconfig manually and add a token field with the copied token returned by AWS CLI, e.g.:

users:
- name: SOMENAME
  user:
    token: PASTE THERE

Then run Kubebox? The token should be picked-up by Kubebox.

[vgibilmanno]

@astefanutti Doesn't work. The token displayed in the login screen is not the same as I entered in the kubeconfig. My config file is located under /root/.kube/config

[astefanutti]

@vgibilmanno the token may just be truncated in the UI. Have you changed the KUBECONFIG environment variable or does you /root/.kube/config file contain multiple contexts?

[astefanutti]

@vgibilmanno Can you also just try a second time connecting, by just pressing enter when the login widget displays with the error message?

[vgibilmanno]

@astefanutti I tried, but it doesn't work. I'm getting the "authentication failed" message. The token doesn't seem to be the one I define in the kubeconfig. I looked for some substring (last 5 characters) but it doesn't match

[astefanutti]

@vgibilmanno thanks for the feedback. Would you mind checking you have only one context defined in your kubeconfig file?

[vgibilmanno]

@astefanutti There is only one context defined. It's exactly like the one I posted before with the addition, that tere is a token value inside the user field.

[astefanutti]

@vgibilmanno thanks. May I ask, is there a token filled in the login widget when using the original kubeconfig, before adding the token manually to it? Finally could you double check the cluster URL displayed in the login widget is the correct one?

[vgibilmanno]

@astefanutti When I use the original kubeconfig, there is a token filled in the login widget. The cluster URL displayed in the login widget is correct.

[astefanutti]

@vgibilmanno thanks again. Just one last test if you don't mind (sorry). Could you re-try the same test of adding the token manually to your kubeconfig file AND removing the whole exec section? Then check the token is the same as the one displayed in the login widget?

[vgibilmanno]

@astefanutti Now the correct token is displayed, but the authentication fails. I tried pressing enter/log-in multiple times in series

[astefanutti]

OK thanks a lot! So there is a request Kubebox does and that's not authorised for your user. Let me check and come back to you. Thanks again!

[astefanutti]

@vgibilmanno if you could try executing the following commands and provide the results:

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/api

$ curl -k -v -H "Authorization: Bearer <TOKEN>" https://OMITTED.amazonaws.com/api/v1/namespaces

[vgibilmanno]

@astefanutti All 3 requests fail with the following message

*   Trying SOMEIP:443...
* TCP_NODELAY set
* Connected to SOMESERVER.amazonaws.com (SOMEIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Jan 23 14:39:32 2019 GMT
*  expire date: Nov 17 23:02:45 2020 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56318da783e0)
> GET / HTTP/2
> Host: SOMEHOST.amazonaws.com
> User-Agent: curl/7.66.0
> Accept: */*
> Authorization: Bearer SOMETOKEN
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 401
< audit-id: 6562ed5f-ef44-4d1b-8210-5c98f0c3cea5
< content-type: application/json
< content-length: 165
< date: Thu, 28 Nov 2019 17:23:36 GMT
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host SOMEHOST.amazonaws.com left intact

[astefanutti]

@vgibilmanno thanks, just to be sure, could you try with a fresh token that you've just retrieve with aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE?

[astefanutti]

@johnpoth I see the Authorization header set when I run the command locally. Not sure about how the CA data could interfere.

[astefanutti]

401 means that the provided token cannot be authenticated to a known identity. It's like the token used is not correct. Maybe there is an encoding issue when Kubebox reads from stdout and parse the response returned from the executed AWS CLI command.

[astefanutti]

@vgibilmanno by any chance, would you be able to clone this repository and run Kubebox from source?

[vgibilmanno]

@astefanutti Yes I'm able to 👍

[astefanutti]

@vgibilmanno awesome! You'll need Node.js installed, so if you can:

• Clone this repository: git clone https://github.com/astefanutti/kubebox.git && cd kubebox
• Then install dependencies: npm i
• Add require('fs').writeFileSync('reponse.json', response); above the following line:

  kubebox/lib/auth/exec.js

  Line 52 in 01b28a2

  this.auth_provider.expiry = Date.parse(json.status.expirationTimestamp);

• Then run node index.js
• Finally cat response.json

Thanks again!

[vgibilmanno]

@astefanutti I get the following in the response.json

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1alpha1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2019-11-29T11:07:51Z",
        "token": "SOMETOKEN"
    }
}

[astefanutti]

@vgibilmanno thanks. Could you try running:

$ aws-iam-authenticator verify -i <clusterid> -t SOMETOKEN

by taking SOMETOKEN from response.json.

[vgibilmanno]

@astefanutti When I enter the cluster name in I get the following:

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

When I enter the cluster address in I get the following:

could not verify token: input token was not properly formatted: non-whitelisted query parameter "X-Amz-SignedHeade"

[astefanutti]

@vgibilmanno could you try:

$ aws-iam-authenticator verify -i SOMEOTHERNAME -t SOMETOKEN

and provide the output of:

$ env

I suspect there are some environment variables that come into the play.

[vgibilmanno]

@astefanutti So the command again fails with:

could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

And have the following entries in env:

HOSTNAME=e39ab07991c2
PYTHON_PIP_VERSION=19.3.1
SHLVL=2
HOME=/root
GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568
PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/ffe826207a010164265d9cc807978e3604d18ca0/get-pip.py
TERM=xterm
PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LANG=C.UTF-8
PYTHON_VERSION=3.8.0
PWD=/
PYTHON_GET_PIP_SHA256=b86f36cc4345ae87bfd4f10ef6b2dbfa7a872fbff70608a1e43944d283fd0eee

[astefanutti]

Damn, would you mind now trying:

$ aws-iam-authenticator verify -i SOMEOTHERNAME -t `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`

(if you don't have jq, you can copy the token manually)

[vgibilmanno]

@astefanutti
&{ARN:arn:aws:sts::SOMEROLEID:assumed-role/SOMEROLE/SOMESESSIONNAME CanonicalARN:arn:aws:iam::SOMEROLEID:role/SOMEROLE AccountID:SOMEACCOUNTID UserID:SOMEUSERID SessionName:SOMESESSIONNAME}

[astefanutti]

@vgibilmanno could you check any difference between the token from response.json and the one returned by aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE?

[vgibilmanno]

@astefanutti The tokens are different. The first 190 characters are identical. The next some thousand characters are different.

[astefanutti]

I don't understand why it does not work when a token generated with AWS CLI, that verifies correctly, is copied into the kubeconfig file and used by Kubebox. And why a token generated within Kubebox does not verify!

I've stumbled upon kubernetes-sigs/aws-iam-authenticator#157. I'm not sure if that applies to that issue to some extent.

[astefanutti]

@vgibilmanno could you try:

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api/v1/namespaces/SOMENAMESPACE/pods

[vgibilmanno]

@astefanutti This one worked!

*   Trying SOMEIP:443...
* TCP_NODELAY set
* Connected to OMMITED.amazonaws.com (SOMEIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=kube-apiserver
*  start date: Jan 23 14:39:32 2019 GMT
*  expire date: Nov 17 23:02:45 2020 GMT
*  issuer: CN=kubernetes
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5653cfda06e0)
> GET /api/v1/namespaces/SOMENAMESPACE/pods HTTP/2
> Host: OMMITED.amazonaws.com
> User-Agent: curl/7.66.0
> Accept: */*
> Authorization: Bearer SOMETOKEN
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< audit-id: c62fa312-7120-426b-970e-9663a90ec01f
< content-type: application/json
< date: Fri, 29 Nov 2019 15:58:25 GMT
<
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/SOMENAMESPACE/pods",
    "resourceVersion": "82672454"
  },
  "items": [SOMEITEMS]
* Connection #0 to host OMMITED.amazonaws.com left intact

[astefanutti]

@vgibilmanno, good, what about:

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api/v1/namespaces

[vgibilmanno]

@astefanutti Only the 2. command worked. The other fails with statuscode 403 Forbidden. The 2. command has the following response body:

"kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "OMITTED.compute.internal:443"
    }
  ]

I tried the following command and it worked too returning a huge json object:

$ curl -k -v -H "Authorization: Bearer aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token" https://OMITTED.amazonaws.com/api/v1

[astefanutti]

@vgibilmanno thanks a lot. Can you confirm the following command returns 403:

$ curl -k -v -H "Authorization: Bearer `aws-iam-authenticator token -i SOMEOTHERNAME -r arn:aws:iam::SOMEROLEID:role/SOMEROLE | jq -r .status.token`" https://OMITTED.amazonaws.com/api/v1/namespaces

[vgibilmanno]

@astefanutti Yes

HTTP/2 403
< audit-id: 31bb7393-cefd-414b-8956-898135d0da83
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 347
< date: Fri, 29 Nov 2019 16:56:30 GMT
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "namespaces is forbidden: User \"SOMEUSER\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope",
  "reason": "Forbidden",
  "details": {
    "kind": "namespaces"
  },
  "code": 403
* Connection #0 to host OMITTED.amazonaws.com left intact

[astefanutti]

OK so I think we've nail down the root cause of this issue. Your user account is not granted permission to list namespaces.

As as work-around, could you update your kubeconfig file and add the namespace you have access to, e.g.:

- context:
    cluster: SOMENAME
    user: SOMENAME
    namespace: NAMESPACE
...

Then run Kubebox.

[vgibilmanno]

@astefanutti ok yes now it works. But it seems like I don't have permissions to see the resources usage metrics. I can execute kubectl top pod though. Well... better than nothing :) I hope I didn't waste too much of your time.

[astefanutti]

@vgibilmanno great! thanks for the feedback. Resources usage metrics requires extra permissions to proxy nodes.

Thanks a lot for your collaboration on this. I think we can let that issue open so that we improve the error message when the user is granted permission to list the namespaces and no namespace is provided.

[bradam12]

I'm 99% sure this issue will cover this but I get an ugly error when I try and list namespaces. I only have access to a single namespace, and it works fine otherwise, but if I hit N to list namespaces this comes up. I understand why it won't let me, but catching the error with a better message would be cool.

[astefanutti]

@bradamson thanks a lot for the feedback. I agree with your suggestion to catch the error and display a proper message in the namespaces list box, instead of dumping the stack trace.