From 9b3f8c9145770ddf57706b672be91db51391778d Mon Sep 17 00:00:00 2001 From: Luuk van Venrooij Date: Wed, 23 Oct 2019 12:03:06 +0200 Subject: [PATCH 1/4] Added design document for modular cli. --- docs/design-docs/modular-cli/modular-cli.png | 3 + docs/design-docs/modular-cli/moduler-cli.md | 177 +++++++++++++++++++ 2 files changed, 180 insertions(+) create mode 100644 docs/design-docs/modular-cli/modular-cli.png create mode 100644 docs/design-docs/modular-cli/moduler-cli.md diff --git a/docs/design-docs/modular-cli/modular-cli.png b/docs/design-docs/modular-cli/modular-cli.png new file mode 100644 index 0000000000..a638f51402 --- /dev/null +++ b/docs/design-docs/modular-cli/modular-cli.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:07dfa44e78fea49a85c6d339c537c7cb4d06334c8d707deac949cc6f878eefd1 +size 91618 diff --git a/docs/design-docs/modular-cli/moduler-cli.md b/docs/design-docs/modular-cli/moduler-cli.md new file mode 100644 index 0000000000..4caeb9fcbb --- /dev/null +++ b/docs/design-docs/modular-cli/moduler-cli.md @@ -0,0 +1,177 @@ +# Epicli modular design document + +Affected version: 0.4.x + +## Goals + +Make epicli easier to work on with multiple teams and make it easier to maintain/extend by: + +1. Splitting up the monotithic Epicli into seperate modules which can run as standalone CLI tools or be linked together through Epicli. +2. Create an extendable plug and play system for roles which can be assigned to components based on certain tasks: apply, upgrade, backup, restore, test etc + +## Architectural design + +The current monitlithic epicli will be split up into the following modules. + +![Module cli design proposal](modular-cli.png) + +### Core + +Shared code between other modules and not executable as standalone. Responsible for: + +- Config +- Logging +- Helpers +- Data schema handling: Loading, defaults, validating etv. +- Build output handling: Loading, saving, updating etc. +- Ansible runner + +### Infrastructure + +Module for creating/destroying cloud infrastructure on AWS/Azure/Google... + "Analysing" existing infrastructure. Maybe at a later time we want to split up the different cloud providers into plugins as well. + +Functionality (rough outline and subjected to change): + +1. template: + ``` + "epicli infra template -f outfile.yaml -p awz/azure/google/any (--all)" + "infra template -f outfile.yaml -p awz/azure/google/any (--all)"? + "Infrastructure.template(...)" + Task: Generate a template yaml with epiphany-cluster definition + possible infra docs when --all is defined + Input: File to output data, provider and possible all flag + Output: outfile.yaml template + ``` +2. apply: + ``` + "epicli infra apply -f data.yaml" + "infra apply -f data.yaml"? + "Infrastructure.apply(...)" + Task: Create/Update infrastucture on AWS/Azure/Google... + Input: Yaml with at least epiphany-cluster + possible infra docs + Output: manifest, ansible inventory and terrafrom files + ``` +3. analyse: + ``` + "epicli infra analyse -f data.yaml" + "infra analyse -f data.yaml"? + "Infrastructure.analyse(...)" + Task: Analysing existing infrastructure + Input: Yaml with at least epiphany-cluster + possible infra docs + Output: manifest, ansible inventory + ``` +4. destroy: + ``` + "epicli infra destroy -b /buildfolder/" + "infra destroy -b /buildfolder/"? + "Infrastructure.destroy(...)" + Task: Destroy all infrastucture on AWS/Azure/Google? + Input: Build folder with manifest and terrafrom files + Output: Deletes the build folder. + ``` + +### Repository + +Module for creating and tearing down a repo + preparing requirements for offline installation. + +Functionality (rough outline and subjected to change): + +1. template: + ``` + "epicli repo template -f outfile.yaml (--all)" + "repo template -f outfile.yaml (--all)"? + "Repository.template(...)" + Task: Generate a template yaml for a repository + Input: File to output data, provider and possible all flag + Output: outfile.yaml template + ``` +2. prepare: + ``` + "epicli repo prepare -os (ubuntu-1904/redhat-7/centos-7)" + "repo prepare -o /outputdirectory/"? + "Repo.prepare(...)" + Task: Create the scripts for downloading requirements for a repo for offline installation for a certain OS. + Input: Os which we want to output the scripts for: (ubuntu-1904/redhat-7/centos-7) + Output: Outputs the scripts scripts + ``` +3. create: + ``` + "epicli repo create -b /buildfolder/ (--offline /foldertodownloadedrequirements)" + "repo create -b /buildfolder/"? + "Repo.create(...)" + Task: Create the repository on a machine (either by running requirement script or copying already prepared ) and sets up the other VMs/machines to point to said repo machine. (Offline and offline depending on --offline flag) + Input: Build folder with manifest and ansible inventory and posible offline requirements folder for onprem installation. + Output: repository manifest or something only with the location of the repo? + ``` +4. teardown: + ``` + "epicli repo teardown -b /buildfolder/" + "repo teardown -b /buildfolder/"? + "Repo.teardown(...)" + Task: Disable the repository and resets the other VMs/machines to their previous state. + Input: Build folder with manifest and ansible inventory + Output: - + ``` + +### Components + +Module for applying a command on a component which can contain one or multiple roles. It will take the Ansible inventory to determin which roles should be applied to which component. The command each role can implement are (rough outline and subjected to change): + +- apply: Command to install roles for components +- backup: Command to backup roles for components +- restore: Command to backup roles for components +- upgrade: Command to upgrade roles for components +- test: Command to upgrade roles for components + +The `apply` command should be implemented for every role but the rest is optional. From an implementation perspective each role will be just a seperate folder inside the plugins directory inside the `components` module folder with command folders which will contain the ansible tasks: + +``` +components-| + |-plugins-| + |-master-| + | |-apply + | |-backup + | |-restore + | |-upgrade + | |-test + | + |-node-| + | |-apply + | |-backup + | |-restore + | |-upgrade + | |-test + | + |-kafka-| + | |-apply + | |-upgrade + | |-test +``` + +Based on the Ansible inventory and the command we can easily select which roles to apply to which components. For the commands we probably also want to introduce some extra flags to only execute commands for certain components. + +Finally we want to add support for an external plugin directory where teams can specify there own role plguins which are not (jet) available inside Epiphany itself. A feature that can also be used by other teams to more easily start contributing developing new components. + +### Epicli + +Bundles all executable modules (Infrastructure, Repository, Component) and adds functions to chain them together: + +Functionality (rough outline and subjected to change): + +1. template: + ``` + "epicli template -f outfile.yaml -p awz/azure/google/any (--all)" + "Epicli.template(...)" + Task: Generate a template yaml with epiphany-cluster definition + possible infrastrucure, repo and component configurations + Input: File to output data, provider and possible all flag + Output: outfile.yaml with templates + ``` +2. apply: + ``` + "epicli apply -f input.yaml" + "Epicli.template(...)" + Task: Sets up a cluster from start to finish + Input: File to output data, provider and possible all flag + Output: Build folder with manifest, ansible inventory, terrafrom files, component setup. + ``` + +... \ No newline at end of file From b45bb34af2434864adc36379fdb9d8d1755a6328 Mon Sep 17 00:00:00 2001 From: Luuk van Venrooij <11056665+seriva@users.noreply.github.com> Date: Wed, 23 Oct 2019 14:04:39 +0200 Subject: [PATCH 2/4] Update docs/design-docs/modular-cli/moduler-cli.md Co-Authored-By: erzetpe --- docs/design-docs/modular-cli/moduler-cli.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/design-docs/modular-cli/moduler-cli.md b/docs/design-docs/modular-cli/moduler-cli.md index 4caeb9fcbb..e19f1c5888 100644 --- a/docs/design-docs/modular-cli/moduler-cli.md +++ b/docs/design-docs/modular-cli/moduler-cli.md @@ -11,7 +11,7 @@ Make epicli easier to work on with multiple teams and make it easier to maintain ## Architectural design -The current monitlithic epicli will be split up into the following modules. +The current monolithic epicli will be split up into the following modules. ![Module cli design proposal](modular-cli.png) @@ -174,4 +174,4 @@ Functionality (rough outline and subjected to change): Output: Build folder with manifest, ansible inventory, terrafrom files, component setup. ``` -... \ No newline at end of file +... From 78b01343838dc6ba9a84e34a95152841de57a077 Mon Sep 17 00:00:00 2001 From: Luuk van Venrooij <11056665+seriva@users.noreply.github.com> Date: Wed, 23 Oct 2019 14:04:58 +0200 Subject: [PATCH 3/4] Update docs/design-docs/modular-cli/moduler-cli.md Co-Authored-By: erzetpe --- docs/design-docs/modular-cli/moduler-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-docs/modular-cli/moduler-cli.md b/docs/design-docs/modular-cli/moduler-cli.md index e19f1c5888..7ddd388234 100644 --- a/docs/design-docs/modular-cli/moduler-cli.md +++ b/docs/design-docs/modular-cli/moduler-cli.md @@ -114,7 +114,7 @@ Functionality (rough outline and subjected to change): ### Components -Module for applying a command on a component which can contain one or multiple roles. It will take the Ansible inventory to determin which roles should be applied to which component. The command each role can implement are (rough outline and subjected to change): +Module for applying a command on a component which can contain one or multiple roles. It will take the Ansible inventory to determine which roles should be applied to which component. The command each role can implement are (rough outline and subjected to change): - apply: Command to install roles for components - backup: Command to backup roles for components From 9a7e67532c6e89e3510923d71f5ced91d0675817 Mon Sep 17 00:00:00 2001 From: Luuk van Venrooij <11056665+seriva@users.noreply.github.com> Date: Wed, 23 Oct 2019 14:05:04 +0200 Subject: [PATCH 4/4] Update docs/design-docs/modular-cli/moduler-cli.md Co-Authored-By: erzetpe --- docs/design-docs/modular-cli/moduler-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design-docs/modular-cli/moduler-cli.md b/docs/design-docs/modular-cli/moduler-cli.md index 7ddd388234..6c6ed22c09 100644 --- a/docs/design-docs/modular-cli/moduler-cli.md +++ b/docs/design-docs/modular-cli/moduler-cli.md @@ -149,7 +149,7 @@ components-| Based on the Ansible inventory and the command we can easily select which roles to apply to which components. For the commands we probably also want to introduce some extra flags to only execute commands for certain components. -Finally we want to add support for an external plugin directory where teams can specify there own role plguins which are not (jet) available inside Epiphany itself. A feature that can also be used by other teams to more easily start contributing developing new components. +Finally we want to add support for an external plugin directory where teams can specify there own role plguins which are not (yet) available inside Epiphany itself. A feature that can also be used by other teams to more easily start contributing developing new components. ### Epicli