Eclipse Che CLI Refactoring + Improvements

[TylerJewell]

This epic covers a number of improvements to refactor the Che CLI to:

1. Allow for offline installation mode.
2. Eliminate duplicate code between che-server, che-launcher, che.sh, CLI.
3. Simplify the CI management of dependent containers.
4. Allow for "custom cli assemblies" so products like OpenShift and Codenvy can reuse + extend CLI.
5. Move utility containers from eclipse/che-dockerfiles to eclipse/che repository.
6. Port the ARTIK IDE to make use of the improved CLI.
7. Add configuration management to Che installations
8. Support docker-compose as an orchestrator.
9. Provide multipe orchestrator launch mechanisms with a single configuration.
10. Simplify configuration of Che into a set of environment variables.
11. Provide backup / restore functionality for Che.

The work for this epic is in the 'cli-refactoring' branch. The order of the specification below indicates the order of work to be done.

CLI Consolidation

We want to consolidate the CLIs that exist between Che and Codenvy, and move all of the meaningful functionality into Che. Most of this work has already been captured in the cli-refactoring branch. The key elements of this work are:

• Support 'che init', "che download', 'che config', 'che start', 'che stop', 'che restart', 'che destroy', 'che offline'.

• Configure the che-server docker container and the docker-compose script template that is provided to have all che-server logs and machine logs written to the /instance/config folder. This would be identical to the way that Codenvy does it, and it is ensuring that the folder mounts of the various orchestrators are done in a proper way.

• Add support for 'che backupandche restore`, to match the cross-platform backup and recovery capabilities that are now reflected in Codenvy.

• Migrate existing CLI function for 'che compile', 'che mount', 'che dir', 'che ssh', 'che test" into the new CLI. We will not be migrating "che profile" into the new CLI. That will die.

• Perform another CLI refactoring to create updated versions of eclipse/che-launcher and codenvy/che-launcher, which provide the important functions of start(), restart(), stop(), config(), init(), version(), backup(), restore(), upgrade(), offline(), download(). This will move the heft of the CLI into a single container, and then make the CLI an optional wrapper. This also has an affect of no longer requiring docker compose 1.8 on the host as we can provide that client in the container. Specification details are at the bottom.

• Need to test: workspace logs written to disk, ssh, sync, dir commands.

Specialized Testing

• The new Che and Codenvy CLI's have not been tested aggressively against systems that have boot2docker, and we think it will likely break. There are a number of situations where we write (or perform tests) in the current directory. We should either just fail with an error message on what to change, or we should detect boot2docker and override CHE_CONFIG, CHE_INSTANCE and any local directory activities into %userprofile%

• Verify that the CLI will work on windows with CHE_CONFIG, CHE_INSTANCE or CLI paths that have dots "." or spaces in their path name.

• Work with the original poster to migrate any imrpovements from CHE-2874 Let users configure the 'Z' flag when mouting a volume #2921 so that this configurability is supported with the CLI.

** merge at this point ** then make more improvements:

Dockerfile Management

We want to move all of the non-stack images into the che repository into the /dockerfiles directory. This also includes the /lib folder which has important DTO utilities. By moving these images into the Che repository, we can tag the code and the images to keep them consistent with releases.

There will be a number of improvements to make:

• Have folders with /dockerfiles/ip/ as a folder name, instead of /dockerfiles/che-ip/.
• We will move the core Che Dockerfiles and Dockerfile.centos into /dockerfiles/che/. This will require updating the Dockerfile to make sure the assembly is referenced in the right location. We are also moving /assembly/src/bin/docker.sh into this dockerfiles folder with an entrypoint.sh name.
• Each dockerfiles directory should have a singular "build.sh " which will create the image such as "eclipse/che-ip:"
• We need to update the comments in the Dockerfile for each image so that the build and run instructions are accurate.
• Update CI systems so that the images can be built and pushed, using these images instead of the ones on che/dockerfiles.
• We should move any files that are to be COPY or ADD into an image as a subdirectory of the Dockerfile directory. For example the /dockerfiles/version/ folder will have the Dockerfile and entrypoint for the image. We would then move the /version folder which has files that are packaged into that image. This would also mean that we might modify the assembly/assembly-main to generate assemblies not only in /target, but also place a copy into /dockerfiles/che/ as well since that image requires an assembly.
• Clean up the CLI docs to only reference this new approach. We would still document the che-launcher and the che-server approach. We would eliminate any usage of Vagrant.

** merge at this point ** then make more improvements:

Create Custom CLI for each Assembly

• Port the new eclipse/che-cli to ARTIK.
• Port the new codenvy/cli to Codenvy SAAS. See Port Che CLI for Codenvy SaaS codenvy/codenvy#1340

** merge at this point ** then make more improvements:

Improve the new Che CLI format for configuration management

• There is now a new che.env file which is a single point of configuration. We should extend che.pp and the che.properties.erb files to allow for more environmental configuration to cover the full range of Che customizations that are possible. For example, the Codenvy CLI now allows for customizing private docker registries with this format, so add this capability in here.
• Similar to the docker-compose.yml, we should also generate for each instance a bash script that will run the che-launcher without using docker-compose.yml.

Eliminate functional redundancies

• We have in some places the same function for bash repeated in four places - in che-launcher, the che CLI, the che-server entrypoint, and the codenvy CLI. Yuck, yuck, and yuck. The CLI is further complicated by the fact that we have che.sh and cli.sh, a separation of two files which is necessary so that the CLI can be self-updating. We want to move all common functions for the CLI into /dockerfiles/lib file. And then we need a way to automate the generation of scripts that depend upon these common functions. The common functions cannot be 'source ' because the CLI needs to embed all of the functions within its file to be sent to customers. So we will need a way to generate codenvy.sh, for example, which sources the functions it needs and adds them into the file below a line that should never be modified. We will then need to test che-server, che-launcher, CLI for Codenvy and Che.

• We then need to build a way to have "CLI Assemblies", which allows different products to reuse certain parts without having to have duplicate methods. While in some cases there are identical functions that can be placed into /lib, there are also similar methods which are not duplicates such as cmd_start(), which are mostly the same, but somewhat different. We will want to move this into an abstraction as a general pupose method which is reusable and extensible at the same time. Tyler will work with engineer to show examples of this. The CI system should allow for the base abstraction to be updated, and immediately reusable for any CLI that wants to clone / reuse it + extend it.

• The Tomcat startup script, che.sh, has a number of Docker-related checks which are no longer required. We should remove all of these checks and optimize the script to only support starting Tomcat. It should be possile to start the Tomcat without assuming any Docker requirements, as we will be supporting localhost machines alongsize docker machines.

• After docs are finished, retag eclipse/che to eclipse/che-cli. When we do this, then remove all evidence of eclipse/che-launcher from docs and DockerHub.

[benoitf]

about item

1. Move utility containers from eclipse/che-dockerfiles to eclipse/che repository.

Does it include to move core typescript library to eclipse/che repository ? (as we're using DTO if will be better to be as part of che repository (same lifecycle))

[TylerJewell]

Yes, that is the goal - even the /lib folder should go into the Che repo.

[TylerJewell]

Closing the epic as finalized. The remaining issue on retagging can be addressed after M9.