feat: add a devworkspace generator tool to replace editor's handlers and support container contribution

[svor]

What does this PR do?

Adds devworkspace-generator tooling into che-devfile-registry to generate DevWorksapces that will be included into the resulted image and support container contribution feature. This generator tool replaces handlers from che-code and che-theia repositories.

For now it generates DevWorkspace with DevWorkspaceTemplate for each sample in the registry. Each sample will have two generated files (one with Theia editor and another one with VSCode editor). The content of the generated files looks like:

devworkspace-che-code-insiders.yaml

apiVersion: workspace.devfile.io/v1alpha2
kind: DevWorkspaceTemplate
metadata:
  name: che-code-golang-example
spec:
  commands:
    - id: init-container-command
      apply:
        component: che-code-injector
  events:
    preStart:
      - init-container-command
  components:
    - name: che-code-runtime-description
      container:
        image: >-
          quay.io/devfile/universal-developer-image@sha256:d1709bbdfa076474f58f796026a2ed2dd3b24fea7e51ce2cc984e729663ff62c
        command:
          - /checode/entrypoint-volume.sh
        volumeMounts:
          - name: checode
            path: /checode
        memoryLimit: 1024Mi
        memoryRequest: 256Mi
        cpuLimit: 500m
        cpuRequest: 30m
        endpoints:
          - name: che-code
            attributes:
              type: main
              cookiesAuthEnabled: true
              discoverable: false
              urlRewriteSupported: true
            targetPort: 3100
            exposure: public
            path: '?tkn=eclipse-che'
            secure: false
            protocol: https
          - name: code-redirect-1
            attributes:
              discoverable: false
              urlRewriteSupported: true
            targetPort: 13131
            exposure: public
            protocol: http
          - name: code-redirect-2
            attributes:
              discoverable: false
              urlRewriteSupported: true
            targetPort: 13132
            exposure: public
            protocol: http
          - name: code-redirect-3
            attributes:
              discoverable: false
              urlRewriteSupported: true
            targetPort: 13133
            exposure: public
            protocol: http
      attributes:
        app.kubernetes.io/component: che-code-runtime
        app.kubernetes.io/part-of: che-code.eclipse.org
        controller.devfile.io/container-contribution: true
    - name: checode
      volume: {}
    - name: che-code-injector
      container:
        image: quay.io/che-incubator/che-code:insiders
        command:
          - /entrypoint-init-container.sh
        volumeMounts:
          - name: checode
            path: /checode
        memoryLimit: 128Mi
        memoryRequest: 32Mi
        cpuLimit: 500m
        cpuRequest: 30m
---
apiVersion: workspace.devfile.io/v1alpha2
kind: DevWorkspace
metadata:
  name: golang-example
spec:
  started: true
  template:
    components:
      - name: tools
        container:
          image: quay.io/devfile/universal-developer-image:ubi8-0e189d9
          env:
            - name: GOPATH
              value: /projects:/home/user/go
            - name: GOCACHE
              value: /tmp/.cache
          endpoints:
            - name: 8080-tcp
              targetPort: 8080
          memoryLimit: 2Gi
          mountSources: true
        attributes:
          controller.devfile.io/merge-contribution: true
    commands:
      - id: run-outyet
        exec:
          label: 1.1 Run outyet
          component: tools
          workingDir: ${PROJECT_SOURCE}/outyet
          commandLine: go get -d && go run main.go
          group:
            kind: run
      - id: stop-outyet
        exec:
          label: 1.2 Stop outyet
          component: tools
          commandLine: kill $(pidof go)
          group:
            kind: run
      - id: test-outyet
        exec:
          label: 1.3 Test outyet
          component: tools
          workingDir: ${PROJECT_SOURCE}/outyet
          commandLine: go test
          group:
            kind: test
      - id: run-current-file
        exec:
          label: Run current file
          component: tools
          workingDir: ${fileDirname}
          commandLine: go get -d && go run ${file}
          group:
            kind: run
    projects:
      - name: golang-example
        zip:
          location: '{{ INTERNAL_URL }}/resources/v2/golang-example.zip'
  contributions:
    - name: editor
      kubernetes:
        name: che-code-golang-example

devworkspace-che-theia-latest.yaml

apiVersion: workspace.devfile.io/v1alpha2
kind: DevWorkspaceTemplate
metadata:
  name: che-theia-golang-example
spec:
  commands:
    - id: init-container-command
      apply:
        component: remote-runtime-injector
  events:
    preStart:
      - init-container-command
  components:
    - name: theia-ide
      container:
        image: >-
          quay.io/eclipse/che-theia@sha256:5db3dfe292bb9e0dd01b652626db9a916b2ddb83add2a143e4f604168dc601fb
        env:
          - name: THEIA_PLUGINS
            value: local-dir:///plugins
          - name: HOSTED_PLUGIN_HOSTNAME
            value: 0.0.0.0
          - name: HOSTED_PLUGIN_PORT
            value: '3130'
          - name: THEIA_HOST
            value: 127.0.0.1
        volumeMounts:
          - name: plugins
            path: /plugins
          - name: theia-local
            path: /home/theia/.theia
        mountSources: true
        memoryLimit: 512M
        cpuLimit: 1500m
        cpuRequest: 100m
        endpoints:
          - name: theia
            attributes:
              type: main
              cookiesAuthEnabled: true
              discoverable: false
              urlRewriteSupported: true
            targetPort: 3100
            exposure: public
            secure: false
            protocol: https
          - name: webviews
            attributes:
              type: webview
              cookiesAuthEnabled: true
              discoverable: false
              unique: true
              urlRewriteSupported: true
            targetPort: 3100
            exposure: public
            secure: false
            protocol: https
          - name: mini-browser
            attributes:
              type: mini-browser
              cookiesAuthEnabled: true
              discoverable: false
              unique: true
              urlRewriteSupported: true
            targetPort: 3100
            exposure: public
            secure: false
            protocol: https
          - name: theia-dev
            attributes:
              type: ide-dev
              discoverable: false
              urlRewriteSupported: true
            targetPort: 3130
            exposure: public
            protocol: http
          - name: theia-redirect-1
            attributes:
              discoverable: false
              urlRewriteSupported: true
            targetPort: 13131
            exposure: public
            protocol: http
          - name: theia-redirect-2
            attributes:
              discoverable: false
              urlRewriteSupported: true
            targetPort: 13132
            exposure: public
            protocol: http
          - name: theia-redirect-3
            attributes:
              discoverable: false
              urlRewriteSupported: true
            targetPort: 13133
            exposure: public
            protocol: http
          - name: terminal
            attributes:
              type: collocated-terminal
              discoverable: false
              cookiesAuthEnabled: true
              urlRewriteSupported: true
            targetPort: 3333
            exposure: public
            secure: false
            protocol: wss
      attributes:
        app.kubernetes.io/component: che-theia
        app.kubernetes.io/part-of: che-theia.eclipse.org
    - name: plugins
      volume: {}
    - name: theia-local
      volume: {}
    - name: theia-ide-contributions
      container:
        image: >-
          quay.io/devfile/universal-developer-image@sha256:d1709bbdfa076474f58f796026a2ed2dd3b24fea7e51ce2cc984e729663ff62c
        args:
          - sh
          - '-c'
          - ${PLUGIN_REMOTE_ENDPOINT_EXECUTABLE}
        env:
          - name: PLUGIN_REMOTE_ENDPOINT_EXECUTABLE
            value: /remote-endpoint/plugin-remote-endpoint
          - name: THEIA_PLUGINS
            value: local-dir:///plugins/sidecars/tools
        volumeMounts:
          - name: plugins
            path: /plugins
          - name: remote-endpoint
            path: /remote-endpoint
        memoryLimit: 512Mi
      attributes:
        controller.devfile.io/container-contribution: true
    - name: remote-endpoint
      volume:
        ephemeral: true
    - name: che-machine-exec
      container:
        image: >-
          quay.io/eclipse/che-machine-exec@sha256:d7bb6e3fb32004e890b50b49affb417a83ae89006fe84766b0c7316437c239ce
        command:
          - /go/bin/che-machine-exec
          - '--url'
          - 127.0.0.1:3333
        memoryLimit: 128Mi
        memoryRequest: 32Mi
        cpuLimit: 500m
        cpuRequest: 30m
      attributes:
        app.kubernetes.io/component: machine-exec
        app.kubernetes.io/part-of: che-theia.eclipse.org
    - name: remote-runtime-injector
      container:
        image: >-
          quay.io/eclipse/che-theia-endpoint-runtime-binary@sha256:fe58069773a27a983d530523a1a0aadb41e158e33d32fc8b486f8bb926aa1e29
        env:
          - name: PLUGIN_REMOTE_ENDPOINT_EXECUTABLE
            value: /remote-endpoint/plugin-remote-endpoint
          - name: REMOTE_ENDPOINT_VOLUME_NAME
            value: remote-endpoint
        volumeMounts:
          - name: plugins
            path: /plugins
          - name: remote-endpoint
            path: /remote-endpoint
      attributes:
        app.kubernetes.io/component: remote-runtime-injector
        app.kubernetes.io/part-of: che-theia.eclipse.org
---
apiVersion: workspace.devfile.io/v1alpha2
kind: DevWorkspace
metadata:
  name: golang-example
spec:
  started: true
  template:
    components:
      - name: tools
        container:
          image: quay.io/devfile/universal-developer-image:ubi8-0e189d9
          env:
            - name: GOPATH
              value: /projects:/home/user/go
            - name: GOCACHE
              value: /tmp/.cache
          endpoints:
            - name: 8080-tcp
              targetPort: 8080
          memoryLimit: 2Gi
          mountSources: true
        attributes:
          controller.devfile.io/merge-contribution: true
    commands:
      - id: run-outyet
        exec:
          label: 1.1 Run outyet
          component: tools
          workingDir: ${PROJECT_SOURCE}/outyet
          commandLine: go get -d && go run main.go
          group:
            kind: run
      - id: stop-outyet
        exec:
          label: 1.2 Stop outyet
          component: tools
          commandLine: kill $(pidof go)
          group:
            kind: run
      - id: test-outyet
        exec:
          label: 1.3 Test outyet
          component: tools
          workingDir: ${PROJECT_SOURCE}/outyet
          commandLine: go test
          group:
            kind: test
      - id: run-current-file
        exec:
          label: Run current file
          component: tools
          workingDir: ${fileDirname}
          commandLine: go get -d && go run ${file}
          group:
            kind: run
    projects:
      - name: golang-example
        zip:
          location: '{{ INTERNAL_URL }}/resources/v2/golang-example.zip'
  contributions:
    - name: editor
      kubernetes:
        name: che-theia-golang-example

NOTE: One problem I've detected. Che-theia devworkspace-handler library analyzes .vscode/extesnions.json and .che/ce-theia-plugins.json files in the sample repository and adds all Theia plugins into the generated devworkspace, in this case all plugins will be installed in the workspace automatically. With this new generator tool plugins won't be added into the DevWorksapce with Theia editor.

Screenshot/screencast of this PR

What issues does this PR fix or reference?

eclipse-che/che#21832

How to test this PR?

• Deploy freshest Che and apply vsvydenko/che-devfile-registry:contributions image for devfile-registry
• Start any workspace from the existing samples in the Dashboard
• Worksapce should start from the pregenerated resources
• Check DevWorkspace and DevWorksapceTemplate custom resources in the OS console

PR Checklist

As the author of this Pull Request I made sure that:

• The Eclipse Contributor Agreement is valid
• Code produced is complete
• Code builds without errors
• Tests are covering the bugfix
• The repository devfile is up to date and works
• Sections What issues does this PR fix or reference and How to test this PR completed
• Relevant user documentation updated
• Relevant contributing documentation updated
• CI/CD changes implemented, documented and communicated

Reviewers

Reviewers, please comment how you tested the PR when approving it.

[github-actions]

Click here to review and test in web IDE:

[l0rd]

LGTM. I have tested the samples plus I have run a script like this one to spot differences among different objects and everything looks good 👍

I have just a couple of questions:

Since you are pushing on main branch shouldn't you call the devworkspace files devworkspace-che-theia-next.yaml instead of devworkspace-che-theia-latest.yaml? That would be also consistent with che-code-insiders.

And why aren't we generating che-idea based devworkspaces?

#!/bin/bash

set -e -u

SAMPLES=(apache-camel-k apache-camel-springboot bash cpp dotnet-asp.net go java-lombok java-mongo java-web-spring nodejs nodejs-angular nodejs-mongo nodejs-react nodejs-yarn php-symfony python quarkus rust scala-sbt)


for SAMPLE in "${SAMPLES[@]}"
do
	IMAGE='vsvydenko/che-devfile-registry:contributions'
	DWPATH="/usr/local/apache2/htdocs/devfiles/$SAMPLE/devworkspace-che-code-insiders.yaml"
  DWPATH2="/usr/local/apache2/htdocs/devfiles/$SAMPLE/devworkspace-che-theia-latest.yaml"

	docker run -ti --rm --entrypoint="cat" $IMAGE $DWPATH | split -a 1 -p '---' - /tmp/devworkspace
	mv /tmp/devworkspacea /tmp/dwt
	mv /tmp/devworkspaceb /tmp/dw
	
	docker run -ti --rm --entrypoint="cat" $IMAGE $DWPATH2 | split -a 1 -p '---' - /tmp/devworkspace2
	mv /tmp/devworkspace2a /tmp/dwt2
	mv /tmp/devworkspace2b /tmp/dw2

#	diff /tmp/dwt-orig /tmp/dwt || true
	diff /tmp/dw /tmp/dw2 || true
#	cat /tmp/dw | yq .spec.template.components > /tmp/dw-clean


#  GIT_SAMPLE=`cat /tmp/dw | yq '.spec.template.projects[0].name'`
#	DFURL="https://raw.githubusercontent.com/che-samples/$GIT_SAMPLE/devfilev2/devfile.yaml"
#  echo "$DFURL"
#	curl -fsSL $DFURL > /tmp/devfile
	
#	cat /tmp/devfile | yq .components > /tmp/devfile-clean
#	diff /tmp/devfile-clean /tmp/dw-clean || true
done

[github-actions]

Click here to review and test in web IDE:

[svor]

Since you are pushing on main branch shouldn't you call the devworkspace files devworkspace-che-theia-next.yaml instead of devworkspace-che-theia-latest.yaml?

@l0rd che-theia-next editor was removed from the editor's list: eclipse-che/che#21814
now we have only latest

And why aren't we generating che-idea based devworkspaces?

with my last commit devWorkspaces are generated with che-idea-next editor as well