Disabling and addon with CRD borks k8s-client

[jnummelin]

What happened:

Disabled Lens and Storage at same go.

got error (K8s::Error::NotFound): GET /apis/beta.kontena.io/v1/groups => HTTP 404 Not Found: 404 page not found
    /__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/transport.rb:213:in `parse_response'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/transport.rb:265:in `block in requests'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/transport.rb:261:in `map'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/transport.rb:261:in `requests'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/transport.rb:318:in `gets'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/resource_client.rb:52:in `list'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/client.rb:128:in `list_resources'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/stack.rb:124:in `prune'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/k8s-client-0.6.4/lib/k8s/stack.rb:160:in `delete'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.1.3/lib/pharos/addon.rb:247:in `delete_resources'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.1.3/lib/pharos/addon.rb:203:in `apply_uninstall'
/__enclose_io_memfs__/lib/ruby/gems/2.5.0/gems/pharos-cluster-2.1.3/lib/pharos/addon.rb:183:in `apply'

Re-upping fixes as it also resets the k8s-client.

This is really a bug on k8s-client side, it needs to better handle CRD changes with it's API cache.

What you expected to happen:

All addon operations succeeding smoothly.

How to reproduce it (as minimally and precisely as possible):

First up with:

network: {} # Use default network config
addons: # Addons for the cluster
  ingress-nginx:
    enabled: true
  kontena-storage:
    enabled: true
    data_dir: /var/lib/kontena-storage
    storage:
      use_all_nodes: true
    dashboard:
      enabled: true
  kontena-lens:
    enabled: true
    name: 'pharos-tutorial-cluster'
    user_management:
      enabled: true
    persistence:
      enabled: true

Then re-up with:

network: {} # Use default network config
addons: # Addons for the cluster
  ingress-nginx:
    enabled: true
  kontena-storage:
    enabled: false
    data_dir: /var/lib/kontena-storage
    storage:
      use_all_nodes: true
    dashboard:
      enabled: true
  kontena-lens:
    enabled: false
    name: 'pharos-tutorial-cluster'
    user_management:
      enabled: true
    persistence:
      enabled: true

Anything else we need to know?:

Environment:

• Pharos version (use pharos --version): 2.1.3
• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release; both for the client and for the used nodes): N/A
• Others:

cluster.yml: