Helm chart seldon-manager role missing permissions for createResources: true

[wongelz]

When createResources: true in values.yaml, seldon-controller-manager fails to start with this error:

2020-05-28T02:56:24.140Z	ERROR	setup	unable to initialise operator	{"error": "customresourcedefinitions.apiextensions.k8s.io \"seldondeployments.machinelearning.seldon.io\" is forbidden: User \"system:serviceaccount:default:seldon-manager\" cannot get resource \"customresourcedefinitions\" in API group \"apiextensions.k8s.io\" at the cluster scope"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
main.main
	/workspace/main.go:95
runtime.main
	/usr/local/go/src/runtime/proc.go:203

Copying the following from (https://github.com/SeldonIO/seldon-core/blob/master/operator/seldon-operator/deploy/role.yaml) to clusterrole_seldon-manager-role.yaml seems to make it work:

- apiGroups:
  - apiextensions.k8s.io
  resources:
  - customresourcedefinitions
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - get
  - list
  - create
  - update

In fact there are a few more rules that are contained in https://github.com/SeldonIO/seldon-core/blob/master/operator/seldon-operator/deploy/role.yaml but missing in the helm chart.

[ukclivecox]

Is there a particular reason you are setting createResources? If you are just using Helm you can leave it as false. In master we have renamed createResources to managerCreateResources as its purpose is to say the manager container should create the needed resources rather than using yaml. This is needed only in certain restricted environments where the yaml can not be used directly.

There is also a bug fix for createResources in master. So I would suggest not setting it unless you really need to.

[wongelz]

Reason is we've got seldon installed in its own (not default) namespace. When we create the SeldonDeployment we get this error:

Internal error occurred: failed calling webhook "v1alpha2.mseldondeployment.kb.io": Post https://seldon-webhook-service.seldon-system.svc:443/mutate-machinelearning-seldon-io-v1alpha2-seldondeployment?timeout=30s: x509: certificate is valid for seldon-webhook-service.default, seldon-webhook-service.default.svc, not seldon-webhook-service.seldon-system.svc

[ukclivecox]

How did you install? If you use the --namespace argument to Helm it should create the correct certificates:

seldon-core/helm-charts/seldon-core-operator/templates/webhook.yaml

Lines 1 to 5 in e504bf4

	{{- if not .Values.createResources }}
	{{- $altNames := list ( printf "seldon-webhook-service.%s" .Release.Namespace ) ( printf "seldon-webhook-service.%s.svc" .Release.Namespace ) -}}
	{{- $ca := genCA "custom-metrics-ca" 365 -}}
	{{- $cert := genSignedCert "seldon-webhook-service" nil $altNames 365 $ca -}}
	---

[wongelz]

Oh OK. We've been using kustomize to set the namespace.

Thanks for your advice!

[ukclivecox]

For Kustomize you would need to use cert-manager. That should work also.