Update to Operator SDK 1.1

[amisevsk]

Note this PR is a re-do of #186 based off a branch in this repo.

What does this PR do?

Update to Operator SDK 1.1.

This process requires bootstrapping a new project using kubebuilder and porting files over to the new structure. See this repo for a more detailed commit history (though it's still a bit messy and hard to read). I generally tried to avoid making unnecessary changes, to make the transition easier, so the configmap is still used, webhooks are done as they are currently, etc.

Main directory structure changes:

• deploy/ => config/

  • The deployment yaml files are now defined using kustomize.
  • The "main" deployment is built from config/default
  • development deployments are generated from config/devel/
  • CRDs are stored in config/crds/
  • Pieces of the deployment are defined in config/components/ and combined
  • Plugin registry is defined in config/registry
  • RBAC files for the controller are generated from annotations on *_controller.go files

• pkg/apis/ => apis/

  • subdirectores in apis/ are named after the resource's group (controller in our case)

• pkg/controller/ => controllers

  • subdirectories in controllers are named for the group of the CR they control (controller for Components, WorkspaceRoutings; workspace for DevWorkspaces)
  • the controllers/controller subdirectory is further divided to distinguish between the controller for Components and WorkspaceRoutings.

• pkg now contains mostly helper code from pkg in the current repo

Other notable differences

• No longer have a hard dependency on operator-sdk code.
• Makefile had to be reworked significantly; there might be gaps in functionality.
• Setting up controllers is done differently now which makes some things harder (watching a configmap)
• To simplify importing CRDs for devfile/api, I reworked the script to grab the files. It now downloads to a temp dir and copies definitions to the expected directory (config/crds). The copied files are gitignored.

Further TODOs

• Remove the configmap-related code from the controller and use environment variables instead, or figure out proper way to configure an operator.
• Depends on changes to devfile/api, reflected by changes here. This is set in go.mod as a replace.
• I didn't look into adapting e2e tests.

What issues does this PR fix or reference?

Resolves #180

We'll likely need to do additional fixes to finalize this work

Is it tested? How?

Tested on minikube; still need to test WTO and crc use cases

[JPinkney]

I just have some initial comments/questions, I'm going to test it on minikube/crc later today

[sleshchenko]

I was not able to test it on crc yet

[sleshchenko]

e2e tests need context.TODO() to be added in a few places. Check it with make test

[sleshchenko]

That's seems to be too general labels which may overlap with other generated controllers. What if we use here something devworkspace specific?

[amisevsk]

I mostly left it because all kubebuilder projects are generated with this label and I didn't see anything about it in the documentation. We can probably remove it.

[amisevsk]

I've removed the labels now and we're using only the ones from before. I still don't know what the purpose of the label is supposed to be, though.

[amisevsk]

I ran into some strange errors running on OpenShift today; I'm looking into the issue.

• On crc when deployed in cluster, controller loops on creating v1.Role for the workspace. No error is returned but no object is created. Running locally does not reproduce this problem
• Since we can't watch the configmap anymore, on OpenShift controller can't detect autogenerated plugin registry route name
• With OAuth routing, I see an error that the oauth container is duplicated in the pod.

[sleshchenko]

On crc when deployed in cluster, controller loops on creating v1.Role for the workspace. No error is returned but no object is created. Running locally does not reproduce this problem

after adding some additional logs, I got

2020-11-03T12:34:36.481Z        DEBUG   controllers.DevWorkspace        Managing K8s Object     {"Request.Namespace": "devworkspace-controller", "Request.Name": "web-terminal", "kind": "v1.Role", "name": "workspace"}
2020-11-03T12:34:36.481Z        INFO    controllers.DevWorkspace            => Creating v1.Role {"Request.Namespace": "devworkspace-controller", "Request.Name": "web-terminal", "namespace": "devworkspace-controller", "name": "workspace"}
2020-11-03T12:34:36.558Z        DEBUG   controllers.DevWorkspace        created.        {"Request.Namespace": "devworkspace-controller", "Request.Name": "web-terminal", "error": "roles.rbac.authorization.k8s.io \"workspace\" is forbidden: user \"system:serviceaccount:devworkspace-controller:default\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:devworkspace-controller\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"pods/exec\"], Verbs:[\"create\"]}\n{APIGroups:[\"apps\"], Resources:[\"replicasets\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"extensions\"], Resources:[\"deployments\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"extensions\"], Resources:[\"replicasets\"], Verbs:[\"get\" \"list\" \"watch\"]}"}

I've pushed changes to log such error 6ea3cff and going to provide fix for roles as well.

P.S. After my fix:

2020-11-03T12:50:13.075Z        ERROR   controller      Reconciler error        {"reconcilerGroup": "workspace.devfile.io", "reconcilerKind": "DevWorkspace", "controller": "devworkspace", "name": "web-terminal", "namespace": "devworkspace-controller", "error": "roles.rbac.authorization.k8s.io \"workspace\" is forbidden: user \"system:serviceaccount:devworkspace-controller:default\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:devworkspace-controller\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"\"], Resources:[\"pods/exec\"], Verbs:[\"create\"]}\n{APIGroups:[\"apps\"], Resources:[\"replicasets\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"extensions\"], Resources:[\"deployments\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"extensions\"], Resources:[\"replicasets\"], Verbs:[\"get\" \"list\" \"watch\"]}"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:237
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

[amisevsk]

Rebased on master and squashed most fixups. Thanks @sleshchenko for figuring out the RBAC and testing code.

I dropped the commit that added workspace roles; instead I've explicitly added the required roles to the main block of devworkspace_controller.go

Tested on crc:

• Theia-nodejs workspace sample (uses openshift-oauth routing) -- works as expected with a fix for webviews route (container name collision on oauth pods)
• Web Terminal UI flow as regular user (the detection code that enables web terminal works if the controller is installed).
  No issues as far as I can see.

[sleshchenko]

Tested cloudshell workspace on crc and it seems to work fine. So, I believe it should be good to proceed with it. Good work 👍

[davidfestal]

Great work @amisevsk !

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amisevsk, davidfestal, sleshchenko

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [amisevsk,sleshchenko]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

New changes are detected. LGTM label has been removed.