Merge pull request #1096 from seriva/additional-documentation

- Added HowTo for defining custom components - Added trouble shooting for out of sync date/time on container - Added some more links for RESOURCES.md - General cleanup and fixes
hitachienergy · Mar 31, 2020 · 746f7cb · 746f7cb
2 parents eb960d2 + 85249cf
commit 746f7cb
Show file tree

Hide file tree

Showing 6 changed files with 126 additions and 52 deletions.
diff --git a/docs/home/GOVERNANCE.md b/docs/home/GOVERNANCE.md
@@ -3,14 +3,14 @@
 <!-- TOC -->
 
 - [Epiphany Governance Model](#epiphany-governance-model)
-    - [Overview](#overview)
-    - [Roles And Responsibilities](#roles-and-responsibilities)
-    - [Committers](#committers)
-    - [Contributors](#contributors)
-    - [Users](#users)
-    - [Support](#support)
-    - [Contribution Process](#contribution-process)
-    - [Decision-Making Process](#decision-making-process)
+  - [Overview](#overview)
+  - [Roles And Responsibilities](#roles-and-responsibilities)
+  - [Committers](#committers)
+  - [Contributors](#contributors)
+  - [Users](#users)
+  - [Support](#support)
+  - [Contribution Process](#contribution-process)
+  - [Decision-Making Process](#decision-making-process)
 
 <!-- /TOC -->
 
@@ -38,15 +38,15 @@ Anyone can become a contributor. There is no expectation of commitment to the pr
 
 Some contributors will already be engaging with the project as users, but will also find themselves doing one or more of the following:
 
-* supporting new users (current users often provide the most effective new user support)
-* reporting bugs
-* identifying requirements
-* supplying graphics and web design
-* programming
-* assisting with project infrastructure
-* writing documentation
-* fixing bugs
-* adding features
+- supporting new users (current users often provide the most effective new user support)
+- reporting bugs
+- identifying requirements
+- supplying graphics and web design
+- programming
+- assisting with project infrastructure
+- writing documentation
+- fixing bugs
+- adding features
 
 As contributors gain experience and familiarity with the project, they may find that the project lead starts relying on them more and more. When this begins to happen, they gradually adopt the role of committer, as described above.
 
@@ -56,10 +56,10 @@ Users are community members who have a need for the project. They are the most i
 
 Users should be encouraged to participate in the life of the project and the community as much as possible. User contributions enable the project team to ensure that they are satisfying the needs of those users. Common user activities may include (but are not limited to):
 
-* evangelizing about the project
-* informing developers of project strengths and weaknesses from a new user’s perspective
-* providing moral support (a ‘thank you’ goes a long way)
-* providing support
+- evangelizing about the project
+- informing developers of project strengths and weaknesses from a new user’s perspective
+- providing moral support (a ‘thank you’ goes a long way)
+- providing support
 
 Users who continue to engage with the project and its community will often find themselves becoming more and more involved. Such users may then go on to become contributors, as described above.
 

diff --git a/docs/home/RESOURCES.md b/docs/home/RESOURCES.md
@@ -4,13 +4,17 @@ Here are some materials concerning Epiphany tooling and cluster components - bot
 
 ## Tooling
 
-1. [Python 3.7](https://docs.python.org/3.7/)
+1. [Visualstudio Code](https://code.visualstudio.com/)
+    - [Devcontainers](https://code.visualstudio.com/docs/remote/containers)
+2. [Python 3.7](https://docs.python.org/3.7/)
     - [Docs and tutorials](https://docs.python.org/3/tutorial/)
-2. [Terraform](https://www.terraform.io/)
-    - AWS use case [example](https://www.terraform.io/intro/getting-started/build.html)
-3. [Ansible](https://www.ansible.com/)
-    - [Intro to playbooks](https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html)
-4. [Azure-cli](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest)
+3. [Terraform](https://www.terraform.io/)
+    - AWS use case [example](https://learn.hashicorp.com/terraform/getting-started/build.html)
+    - Azure use case [example](https://learn.hashicorp.com/terraform?track=azure#azure)
+4. [Ansible](https://www.ansible.com/)
+    - [Introduction to playbooks](https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html)
+5. [Azure-cli](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest)
+6. [Boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html)
 
 ## Cluster Components
 
@@ -46,5 +50,9 @@ Here are some materials concerning Epiphany tooling and cluster components - bot
     1. [HaProxy](http://www.haproxy.org/)
 7. Databases
     1. [PostgreSQL](https://www.postgresql.org/docs/)
+        - [Repmng](https://repmgr.org/)
+        - [PGBouncer](https://www.pgbouncer.org/)
+        - [PGPool](https://www.pgpool.net/mediawiki/index.php/Main_Page)
+        - [PGAudit](https://www.pgaudit.org/)
 8. Authentication
     1. [KeyCloak](https://www.keycloak.org/documentation.html)
diff --git a/docs/home/SECURITY.md b/docs/home/SECURITY.md
@@ -10,19 +10,18 @@ this during planning your Epiphany deployment and configuration.
 
 ### Users and roles created by epiphany
 
-By default Epiphany is creating user operations that is used to connect to machines with admin rights on every machine. This 
-setting can be changed in Epiphany yaml configuration files.
+By default Epiphany is creating user operations that is used to connect to machines with admin rights on every machine. This setting can be changed in Epiphany yaml configuration files.
 
 Additional to users created by each component Epiphany creates also users and groups:
 
-  - haproxy_exporter/haproxy_exporter
-  - kafka_exporter/kafka_exporter
-  - node_exporter/node_exporter
-  - jmx-exporter/jmx-exporter
-  - prometheus/prometheus
-  - rabbitmq/rabbitmq
-  - zookeeper/zookeeper
-  - kafka/kafka
+- haproxy_exporter/haproxy_exporter
+- kafka_exporter/kafka_exporter
+- node_exporter/node_exporter
+- jmx-exporter/jmx-exporter
+- prometheus/prometheus
+- rabbitmq/rabbitmq
+- zookeeper/zookeeper
+- kafka/kafka
 
 Other accounts created by each component you can find in these components documentation.
 

diff --git a/docs/home/TROUBLESHOOTING.md b/docs/home/TROUBLESHOOTING.md
@@ -1,15 +1,32 @@
 # Troubleshooting
 
-## Kubernetes
+## Epicli container connection issues after hibernation/sleep on Windows
 
-Sometimes Google has a connection issue with pulling down images. You may see something like below:
+When running the Epicli container on Windows you might get such errors when trying to run the apply command:
 
-```text
-TASK [master : kubeadm config images pull] **********************************************************************************************
-fatal: [vm-epiphany-rhel-playground-master-001]: FAILED! => {"changed": true, "cmd": "kubeadm config images pull", "delta": "0:00:01.428562", "end": "2018-07-18 08:56:47.608629", "msg": "non-zero return code", "rc": 1, "start": "2018-07-18 08:56:46.180067", "stderr": "failed to pull image \"k8s.gcr.io/kube-apiserver-amd64:v1.11.1\": exit status 1", "stderr_lines": ["failed to pull image \"k8s.gcr.io/kube-apiserver-amd64:v1.11.1\": exit status 1"], "stdout": "", "stdout_lines": []}
+Azure:
+```
+12:28:39 INFO cli.engine.terraform.TerraformCommand - Error: Error reading queue properties for AzureRM Storage Account "cluster": queues.Client#GetServiceProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: error response cannot be parsed: "\ufeff<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:cba2935f-1003-006f-071d-db55f6000000\nTime:2020-02-04T05:38:45.4268197Z</Message><AuthenticationErrorDetail>Request date header too old: 'Fri, 31 Jan 2020 12:28:37 GMT'</AuthenticationErrorDetail></Error>" error: invalid character 'ï' looking for beginning of value
+```
+
+AWS:
+```
+19:50:14 ERROR epicli - An error occurred (AuthFailure) when calling the DescribeImages operation: AWS was not able to validate the provided access credentials
 ```
 
-Wait a little while and try again and it will usually resolve itself quickly. If it does not go away then it could be the version of Kubernetes. For example, in the error above, v1.11.1 did not have proper images in the google registry. Changing to v1.11.0 fixed it until Google fixed their issue.
+These issues might occur when the host machine you are running the Epicli container on was put to sleep or hybernated for an extended period of time. Hyper-V might have issues syncing the time between the container and the host after it wakes up or is resumed. You can confirm this by checking the date and time in your container by running:
+
+```shell
+Date
+```
+
+If the times are out of sync restarting the container will resolve the issue. If you do not want to restart the container you can also run the following 2 commands from an elevated Powershell prompt to force it during container runtime:
+
+```shell
+Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Disable-VMIntegrationService
+
+Get-VMIntegrationService -VMName DockerDesktopVM -Name "Time Synchronization" | Enable-VMIntegrationService
+```
 
 ## Kafka
 

diff --git a/docs/home/howto/CLUSTER.md b/docs/home/howto/CLUSTER.md
@@ -358,18 +358,68 @@ specification:
   - name: auth-service
     enabled: yes # set to yest to enable authentication service
     ... # add other authentication service configuration as needed
----
-kind: configuration/kubernetes-master
-title: Kubernetes Master Config
+
+## How to create custom cluster components
+
+Epiphany gives you the ability to define custom components. This allows you define a custom set of roles for a component you want to use in your cluster and can be usefull when you for example want to maximize usage of the available machines you have at your disposal.
+
+The first thing you will need to do is define it in the `configuration/feature-mapping` configuration. To get this configuration you can run `epicli init ... --full` command. In the `available_roles` roles section you can see all the available roles that Epiphany provides. The `roles_mapping` is where all the Epiphany components are defined and were you need to add your custom components.
+
+Below are parts of an example `configuration/feature-mapping` were we define an new `single_machine_new` component. We want to use Kafka instead of RabbitMQ and don`t need applications and postgress since we dont want a Keycloak deployment:
+
+```yaml
+kind: configuration/feature-mapping
+title: Feature mapping to roles
 name: default
 specification:
-  allow_pods_on_master: true # set to true to enable untaint master for pod deployment
-  ... # add other kubernetes-master configuration as needed
+  available_roles: # All entries here represent the available roles within Epiphany
+  - name: repository
+    enabled: yes
+  - name: firewall
+    enabled: yes
+  - name: image-registry
+  ...
+  roles_mapping: # All entries here represent the default components provided with Epiphany
+  ...
+    single_machine:
+    - repository
+    - image-registry
+    - kubernetes-master
+    - applications
+    - rabbitmq
+    - postgresql
+    - firewall
+    # Below is the new single_machine_new definition
+    single_machine_new:
+    - repository
+    - image-registry
+    - kubernetes-master
+    - kafka
+    - firewall
+  ...
 ```
 
-## How to create custom cluster components
+Once defined the new `single_machine_new` can be used inside the `epiphany-cluster` configuration:
+
+```yaml
+kind: epiphany-cluster
+title: Epiphany cluster Config
+name: default
+specification:
+  prefix: new
+  name: single
+  admin_user:
+    name: operations
+    key_path: /user/.ssh/id_rsa
+  cloud:
+    ... # add other cloud configuration as needed
+  components:
+    ... # other components as needed
+    single_machine_new:
+      count: x
+```
 
-TODO
+*Note: After defining a new component you might also need to define aditional configurations for virtual machines and security rules depending on what you are trying to achieve.*
 
 ## How to scale or cluster components
 
@@ -396,7 +446,7 @@ Then when applying the changed configuration using Epicli additional VM's will b
       is_clustered: true
   ...
   ```
-- postgresql: When changed this will setup or remove additional nodes for Postgresql. Note that extra nodes can only be setup todo replication by adding the following additional `configuration/postgresql` configuration:
+- postgresql: When changed this will setup or remove additional nodes for Postgresql. Note that extra nodes can only be setup to do replication by adding the following additional `configuration/postgresql` configuration:
 
   ```yaml
   kind: configuration/postgresql

diff --git a/docs/home/howto/SECURITY.md b/docs/home/howto/SECURITY.md
@@ -338,7 +338,7 @@ Prerequisites: Epiphany Kubernetes cluster
 
 ## How to run epicli with password
 
-Epiphany encrypts Kubernetes artifacts (access tokens) stored in Epiphany build directory. In order to achieve it, user is asked for password which will be used for encryption and decryption of artifacts. Remember to enter the same password for the same cluster - if password will not be the same, epicli will not be able to decrypt secrets. 
+Epiphany encrypts Kubernetes artifacts (access tokens) stored in Epiphany build directory. In order to achieve it, user is asked for password which will be used for encryption and decryption of artifacts. Remember to enter the same password for the same cluster - if password will not be the same, epicli will not be able to decrypt secrets.
 
 Standard way of executing epicli has not been changed: