Merge pull request hitachienergy#602 from epiphany-platform/develop

Merge develop into master
to-bar · Oct 11, 2019 · c410eb3 · c410eb3
2 parents 516f1c4 + efa5c90
commit c410eb3
Show file tree

Hide file tree

Showing 334 changed files with 12,609 additions and 3,435 deletions.
diff --git a/CHANGELOG-0.4.md b/CHANGELOG-0.4.md
@@ -0,0 +1,27 @@
+# Changelog 0.4
+
+## [0.4.0] 2019-10-11
+
+### Added
+
+- Offline installation
+- Azure cluster deployments with Epicli
+- Delete commands to remove clusters from cloud providers (AWS, Azure)
+- Devcontainer for Epicli development using VSCode
+- Debug flag for Epicli
+
+### Changed
+
+- Various improvements in Epicli
+- Documentation cleanup and updates
+
+### Fixed
+
+- [#407](https://github.com/epiphany-platform/epiphany/issues/407) - Deployment/Application role fails because Kubernetes cluster is not ready after reboot.
+- [#410](https://github.com/epiphany-platform/epiphany/issues/410) - Node_exporter ports are not present in defaults resulting in Prometheus not beeing able to scrape data with minimal cluster data.yaml.
+- [#548](https://github.com/epiphany-platform/epiphany/issues/548) - Epicli fails on AWS when clustering RabbitMQ nodes.
+- [#549](https://github.com/epiphany-platform/epiphany/issues/549) - Need to allow traffic on port 5432 to enable PostgreSQL replication on AWS.
+
+### Known issues
+
+-
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,12 +9,16 @@ Reference for actual cluster component versions can be found [here](docs/home/CO
 
 ## Current release
 
-### 0.3.x
+### 0.4.x
 
-- [CHANGELOG-0.3.0](./CHANGELOG-0.3.md#030-2019-07-31)
+- [CHANGELOG-0.4.0](./CHANGELOG-0.4.md#040-2019-09-30)
 
 ## Older releases
 
+### 0.3.x
+
+- [CHANGELOG-0.3.0](./CHANGELOG-0.3.md#030-2019-07-31)
+
 ### 0.2.x
 
 - [CHANGELOG-0.2.3](./CHANGELOG-0.2.md#023-2019-05-20)

diff --git a/README.md b/README.md
@@ -15,9 +15,16 @@ Epiphany can run on as few as one node (laptop, desktop, server) but the real va
 
 We currently use Terraform and Ansible for our automation orchestration. All automation is idempotent so you can run it as many times as you wish and it will maintain the same state unless you change the data. If someone makes a "snow flake" change to the environment (you should never do this) then simply running the automation again will put the environment back to the desired state.
 
-## Legacy note
+## Note about legacy Epiphany
 
-In Epiphany 0.3 a new CLI tool was introduced (epicli) for deploying and managing clusters and currently supports AWS and bare metal deployment. Azure support will be added soon in a subsequent release but for now if you are in need for deploying a cluster on Azure use the older Legacy engine.
+Epicli 0.4.0 adds support for Azure deployments so using legacy Epiphany is no longer needed. We advice new projects to start straight with Epicli and older projects which still use legacy Epiphany to move over to Epicli 0.4.0. A tool to migrate legacy data files to the new format is in the works and will be released shortly.
+
+The Epicli 0.5.0 release later this year will drop the inclusion of the legacy path entirely.
+
+## Note about documentation
+
+- The documentation is a moving target. Always check the latest documentation on the develop branch. There is a big chance that whatever you are looking for is already added/updated or improved there.
+- We are currently in the process of documenting all features of Epicli and phasing out legacy Epiphany documentation. When documentation is specific for `Epicli` or `Legacy` it will be marked under a header with the these names. If its not under any of these headers then it applies to both.
 
 ## Quickstart
 
@@ -41,7 +48,7 @@ This minimum file definition is fine to start with, if you need more control ove
 epicli init -p aws -n demo --full
 ```
 
-You will need to modify a few values (like you AWS secrets, directory path for ssh keys). Once you are done done with `demo.yaml` you can start cluster deployment by executing with:
+You will need to modify a few values (like your AWS secrets, directory path for SSH keys). Once you are done with `demo.yaml` you can start cluster deployment by executing:
 
 ```shell
 epicli apply -f demo.yaml
@@ -73,16 +80,17 @@ Find more information using table of contents below - especially the [How-to gui
 - Platform
   - [Resources](docs/home/RESOURCES.md)
   - [How-to guides](docs/home/HOWTO.md)
-  - [Troubleshooting](docs/home/TROUBLESHOOTING.md)
+  - [Components](docs/home/COMPONENTS.md)
+  - [Security](docs/home/SECURITY.md)
+  - [Troubleshooting](docs/home/TROUBLESHOOTING.md)  
+  - [Changelog](CHANGELOG.md)  
 - Architecture
   - [Logical View](docs/architecture/logical-view.md)
   - [Process View](docs/architecture/process-view.md)
   - [Physical View](docs/architecture/physical-view.md)
-- Project
-  - [How-to contribute](docs/home/CONTRIBUTING.md)
-  - [Workflow to follow](docs/home/GITWORKFLOW.md)
+- Contributing
   - [Governance model](docs/home/GOVERNANCE.md)
-  - [Components](docs/home/COMPONENTS.md)
-  - [Changelog](CHANGELOG.md)
-
+  - [Development environment](docs/home/DEVELOPMENT.md)
+  - [GIT Workflow](docs/home/GITWORKFLOW.md)
+  
 <!-- TOC -->
diff --git a/core/core/src/ansible/roles/deployments/tasks/main.yml b/core/core/src/ansible/roles/deployments/tasks/main.yml
@@ -1,4 +1,11 @@
 ---
+- name: Wait until the cluster is available
+  shell: kubectl --kubeconfig=/home/{{ admin_user.name }}/.kube/config cluster-info
+  retries: 10
+  delay: 5
+  register: output
+  until: output is succeeded
+
 - name: Include deployments
   include_tasks: "applications/{{ item.name }}/main.yml"
   vars:

diff --git a/core/core/src/ansible/roles/docker/tasks/RedHat.yml b/core/core/src/ansible/roles/docker/tasks/RedHat.yml
@@ -11,7 +11,7 @@
 # Todo: Investigate problems with container selinux on Azure.
 - name: Install container packages
   yum:
-    name: http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.95-2.el7_6.noarch.rpm
+    name: http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.107-1.el7_6.noarch.rpm
     state: present
     update_cache: yes
 

diff --git a/core/core/src/docker/test-CI/Dockerfile b/core/core/src/docker/test-CI/Dockerfile
@@ -1,4 +1,6 @@
-FROM epiphanyregistry.azurecr.io/epiphany-deploy:latest
+ARG BUILD_ID
+
+FROM epiphanyregistry.azurecr.io/epiphany-deploy:$BUILD_ID
 
 RUN mkdir /tmp/keys
 

diff --git a/core/core/src/templates/common/ansible.sh.j2 b/core/core/src/templates/common/ansible.sh.j2
@@ -186,7 +186,7 @@ $REPO_ROOT/bin/template_engine -d $EPIPHANY_DATA_DIR/data/manifest.yaml -i $REPO
 
 echo_yellow '====> Verifying Ansible access to nodes...'
 
-ansible all -i $EPIPHANY_DATA_DIR/inventory/$ANSIBLE_ENV -m ping
+ansible all -i $EPIPHANY_DATA_DIR/inventory/$ANSIBLE_ENV -m ping --timeout=60
 
 if [[ $? -ne 0 ]]; then
     echo_red "ERROR: Unable to reach all of the hosts in the cluster. Verify 'manifest.yaml' is correct and run this again."

diff --git a/core/core/test/serverspec/spec/haproxy_tls_termination/haproxy_tls_termination_spec.rb b/core/core/test/serverspec/spec/haproxy_tls_termination/haproxy_tls_termination_spec.rb
@@ -2,6 +2,7 @@
 
 haproxy_host = 'localhost'
 haproxy_front_port = 443
+haproxy_stats_port = 9000
 
 describe 'Checking if HAProxy service is running' do
   describe service('haproxy') do
@@ -32,8 +33,11 @@
 end
 
 describe 'Checking if the ports are open' do
+  let(:disable_sudo) { false }
   describe port(haproxy_front_port) do
-    let(:disable_sudo) { false }
+    it { should be_listening }
+  end
+  describe port(haproxy_stats_port) do
     it { should be_listening }
   end
 end 
@@ -71,14 +75,14 @@
 end
 
 describe 'Checking HAProxy HTTP status code for stats page' do
-  describe command("curl -k --user $(cat /etc/haproxy/haproxy.cfg | grep 'stats auth' | awk '{print $3}') -o /dev/null -s -w '%{http_code}' \
-  https://#{haproxy_host}:#{haproxy_front_port}$(cat /etc/haproxy/haproxy.cfg | grep 'stats uri' | awk '{print $3}')") do
+  describe command("curl -k --user $(awk '/stats auth/ {print $3}' /etc/haproxy/haproxy.cfg) -o /dev/null -s -w '%{http_code}' \
+  http://#{haproxy_host}:#{haproxy_stats_port}$(awk '/stats uri/ {print $3}' /etc/haproxy/haproxy.cfg)") do
     it "is expected to be equal" do
       expect(subject.stdout.to_i).to eq 200
     end
   end
-  describe command("curl -k --user $(cat /etc/haproxy/haproxy.cfg | grep 'stats auth' | awk '{print $3}') \
-  https://#{haproxy_host}:#{haproxy_front_port}$(cat /etc/haproxy/haproxy.cfg | grep 'stats uri' | awk '{print $3}')") do
+  describe command("curl -k --user $(awk '/stats auth/ {print $3}' /etc/haproxy/haproxy.cfg) \
+  http://#{haproxy_host}:#{haproxy_stats_port}$(awk '/stats uri/ {print $3}' /etc/haproxy/haproxy.cfg)") do
     its(:stdout) { should match /Statistics Report for HAProxy/ }
   end
 end

diff --git a/core/data/azure/infrastructure/epiphany-bld-apps/data.yaml b/core/data/azure/infrastructure/epiphany-bld-apps/data.yaml
@@ -8,7 +8,7 @@
 title: Epiphany Apps Infrastructure...
 
 kind: datafile
-version: 0.3.0
+version: 0.4.0
 
 # NOTE: Any data values that are empty put "" or the value None will be used in the templates for those attributes.
 

diff --git a/core/data/azure/infrastructure/epiphany-playground/basic-data.yaml b/core/data/azure/infrastructure/epiphany-playground/basic-data.yaml
@@ -2,7 +2,7 @@
 # Simplified datafile that you can use together with template (see README.md in this folder).
 # Change values according to your needs, start with generating ssh keys and placing them in the directory "keys_directory". Do not forget to update "keys_directory" as well.
 kind: simplified-datafile
-version: 0.3.0
+version: 0.4.0
 environment_name: Playground
 azure:
   subscription_name: YOUR-SUBSCRIPTION-NAME

diff --git a/core/data/azure/infrastructure/epiphany-qa-basic/basic-data.yaml b/core/data/azure/infrastructure/epiphany-qa-basic/basic-data.yaml
@@ -1,7 +1,7 @@
 ---
 # Simplified datafile that you can use together with QA template.
 kind: simplified-datafile
-version: 0.3.0
+version: 0.4.0
 environment_name: {{ resource_group }}
 azure:
   subscription_name: {{ sp_subscription_name }}

diff --git a/core/data/azure/infrastructure/epiphany-qa-template/data.yaml.j2 b/core/data/azure/infrastructure/epiphany-qa-template/data.yaml.j2
@@ -9,7 +9,7 @@
 title: Epiphany ({{ azure.image_offer }}) {{ environment_name }}
 
 kind: datafile
-version: 0.3.0
+version: 0.4.0
 
 # NOTE: Any data values that are empty put "" or the value None will be used in the templates for those attributes.
 

diff --git a/core/data/azure/infrastructure/epiphany-single-machine/data.yaml b/core/data/azure/infrastructure/epiphany-single-machine/data.yaml
@@ -8,7 +8,7 @@
 title: Epiphany Single Machine Infrastructure...
 
 kind: datafile
-version: 0.3.0
+version: 0.4.0
 
 # NOTE: Any data values that are empty put "" or the value None will be used in the templates for those attributes.
 

diff --git a/core/data/azure/infrastructure/epiphany-template/data.yaml.j2 b/core/data/azure/infrastructure/epiphany-template/data.yaml.j2
@@ -9,7 +9,7 @@
 title: Epiphany ({{ azure.image_offer }}) {{ environment_name }}
 
 kind: datafile
-version: 0.3.0
+version: 0.4.0
 
 # NOTE: Any data values that are empty put "" or the value None will be used in the templates for those attributes.
 

diff --git a/core/data/metal/epiphany-lab/data.yaml b/core/data/metal/epiphany-lab/data.yaml
@@ -4,7 +4,7 @@
 
 
 kind: datafile
-version: 0.3.0
+version: 0.4.0
 
 # This will apply to a VPN like environment or an air-gapped like environment
 bastian:

diff --git a/core/data/metal/epiphany-single-machine/data.yaml b/core/data/metal/epiphany-single-machine/data.yaml
@@ -4,7 +4,7 @@
 
 
 kind: datafile
-version: 0.3.0
+version: 0.4.0
 
 # This will apply to a VPN like environment or an air-gapped like environment
 bastian:

diff --git a/core/manifest.yaml b/core/manifest.yaml
@@ -3,7 +3,7 @@
 # This data is for the core of Epiphany and not the data for a given environment.
 
 name: Epiphany
-version: 0.3.0
+version: 0.4.0
 
 # Set the proxy info up if your environment requires it. This is sometimes the case for on-premise builds/installs
 proxy:

diff --git a/core/src/epicli/.devcontainer/Dockerfile b/core/src/epicli/.devcontainer/Dockerfile
@@ -0,0 +1,35 @@
+FROM python:3.7
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG USERNAME=vscode
+ARG USER_UID=1000
+ARG USER_GID=$USER_UID
+
+RUN apt-get update \
+    && apt-get -y install --no-install-recommends apt-utils dialog 2>&1 \
+
+    && apt-get -y install git procps lsb-release gcc make musl-dev libffi-dev tar unzip \
+
+    && apt-get -y install ruby-full \
+
+    && gem install serverspec rake rspec_junit_formatter \
+
+    && pip --disable-pip-version-check --no-cache-dir install pylint \
+
+    && pip --disable-pip-version-check --no-cache-dir install pipenv \
+
+    && groupadd --gid $USER_GID $USERNAME \
+    && useradd -s /bin/bash --uid $USER_UID --gid $USER_GID -m $USERNAME \
+
+    && apt-get install -y sudo \
+    && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME \
+    && chmod 0440 /etc/sudoers.d/$USERNAME \
+
+    && apt-get autoremove -y \
+    && apt-get clean -y \
+    && rm -rf /var/lib/apt/lists/*
+
+ENV DEBIAN_FRONTEND=
+
+
diff --git a/core/src/epicli/.devcontainer/devcontainer.json b/core/src/epicli/.devcontainer/devcontainer.json
@@ -0,0 +1,18 @@
+{
+	"name": "epicli",
+	"dockerFile": "Dockerfile",
+	"extensions": [
+		"ms-python.python",
+		"littlefoxteam.vscode-python-test-adapter",
+		"vscoss.vscode-ansible",
+		"wholroyd.jinja",
+		"redhat.vscode-yaml",
+		"mauve.terraform",
+		"davidanson.vscode-markdownlint"
+	],
+	"settings": {
+		"python.pythonPath": "/usr/local/bin/python",
+	},
+	"postCreateCommand": "sudo pipenv install --system --dev",
+	"runArgs": [ "-u", "vscode" ]
+}
diff --git a/core/src/epicli/.gitignore b/core/src/epicli/.gitignore
@@ -28,7 +28,7 @@ share/python-wheels/
 *.egg
 MANIFEST
 
-requirements.txt
+/requirements.txt
 
 # PyInstaller
 #  Usually these files are written by a python script from a template
@@ -121,8 +121,11 @@ dmypy.json
 # pycharm settings
 .idea/
 
-# external packages for BDS scan
+# epicli specific
 external/
-
-# test results
-tests_result/
+tests/serverspec-cli/results/
+tests/cli/results/
+.terraform
+clusters
+.vscode
+.devcontainer
diff --git a/core/src/epicli/.vscode/extensions.json b/core/src/epicli/.vscode/extensions.json
@@ -0,0 +1,12 @@
+{
+    "recommendations": [
+        "ms-vscode-remote.remote-containers",
+        "ms-python.python",
+        "littlefoxteam.vscode-python-test-adapter",
+        "vscoss.vscode-ansible",
+        "wholroyd.jinja",
+        "redhat.vscode-yaml",
+        "mauve.terraform",
+        "davidanson.vscode-markdownlint"
+    ]
+}
diff --git a/core/src/epicli/.vscode/launch.json b/core/src/epicli/.vscode/launch.json
@@ -0,0 +1,41 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "epicli",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/cli/epicli.py",
+            "cwd": "${workspaceFolder}",      
+            "pythonPath": "${config:python.pythonPath}",
+            "env": { "PYTHONPATH": "${workspaceFolder}" },
+            "console": "integratedTerminal",
+            "args": ["apply",  "-f",  "${workspaceFolder}/PATH_TO_YOUR_DATA_YAML"]
+        },
+        {
+            "name": "python unit tests",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/run-tests.py",
+            "cwd": "${workspaceFolder}",      
+            "pythonPath": "${config:python.pythonPath}",
+            "env": { "PYTHONPATH": "${workspaceFolder}" },
+            "console": "integratedTerminal",
+            "args": ["python"]
+        },
+        {
+            "name": "server spec tests",
+            "type": "python",
+            "request": "launch",
+            "program": "${workspaceFolder}/run-tests.py",
+            "cwd": "${workspaceFolder}",      
+            "pythonPath": "${config:python.pythonPath}",
+            "env": { "PYTHONPATH": "${workspaceFolder}" },
+            "console": "integratedTerminal",
+            "args": ["spec", "-i", "${workspaceFolder}/PATH_TO_CLUSTER_INVENTORY", "-u", "ADMIN_USER", "-k", "${workspaceFolder}/PATH_TO_SSH_KEY"]
+        }
+    ]	
+}