Merge remote-tracking branch 'upstream/develop' into develop

.github/ISSUE_TEMPLATE/bug_report.md → .github/ISSUE_TEMPLATE/bug-report.md

@@ -13,7 +13,7 @@ A clear and concise description of what the bug is.
 **How to reproduce**
 Steps to reproduce the behavior:
 1. execute `epicli init ... (with params)`
-2. edit config file 
+2. edit config file
 3. execute `epicli apply ...`
 
 **Expected behavior**
@@ -23,8 +23,24 @@ A clear and concise description of what you expected to happen.
 If applicable, add config files to help explain your problem.
 
 **Environment**
-- Cloud provider: [AWS | Azure | Any]
+- Cloud provider: [AWS | Azure | All | None]
 - OS: [e.g. Ubuntu 18.04.4 LTS, you can use `cat /etc/os-release`]
 
+**epicli version**: [`epicli --version`]
+
 **Additional context**
 Add any other context about the problem here.
+
+---
+
+**DoD checklist**
+
+* [ ] Changelog updated (if affected version was released)
+* [ ] COMPONENTS.md updated / doesn't need to be updated
+* [ ] Automated tests passed (QA pipelines)
+  * [ ] apply
+  * [ ] upgrade
+* [ ] Case covered by automated test (if possible)
+* [ ] Idempotency tested
+* [ ] Documentation updated / doesn't need to be updated
+* [ ] All conversations in PR resolved

.github/ISSUE_TEMPLATE/feature-request.md

@@ -18,3 +18,19 @@ A clear and concise description of any alternative solutions or features you've
 
 **Additional context**
 Add any other context or screenshots about the feature request here.
+
+---
+
+**DoD checklist**
+
+* [ ] Changelog updated
+* [ ] COMPONENTS.md updated / doesn't need to be updated
+* [ ] Feature has automated tests
+* [ ] Automated tests passed (QA pipelines)
+  * [ ] apply
+  * [ ] upgrade
+* [ ] Idempotency tested
+* [ ] Documentation added / updated / doesn't need to be updated
+* [ ] All conversations in PR resolved
+* [ ] Solution meets requirements and is done according to design doc
+* [ ] Usage compliant with license

.github/ISSUE_TEMPLATE/release-version.md

@@ -0,0 +1,32 @@
+---
+name: Release version
+about: Prepare and publish a release
+title: "[RELEASE] vX.Y.Z"
+labels: type/release
+assignees: ''
+
+---
+
+#### Release checklist
+
+* [ ] Default OS cloud images updated
+* [ ] Support branch created
+* [ ] CI pipelines for support branch created
+* All automated tests passed (QA pipelines)
+  * [ ] Standard apply (new cluster)
+  * [ ] Standard apply (existing cluster)
+  * [ ] Upgrade for V-1
+  * [ ] Upgrade for V-2
+  * [ ] Upgrade for V-3
+  * [ ] Upgrade for V-4
+  * [ ] k8s HA apply
+  * [ ] Promote k8s to HA apply
+  * [ ] Single machine apply
+  * [ ] Offline mode apply
+* [ ] Changelog updated (release date, known issues, breaking changes, deprecations)
+* [ ] Release pipeline performed
+* [ ] GitHub release exists with correct content (links)
+* [ ] `docker pull` command from the release description tested
+* [ ] Version bumped to the next in develop branch (in version.txt.py)
+* [ ] Changelog for the next version created in develop branch
+* [ ] Releases updated in ZenHub

CHANGELOG-0.10.md

@@ -4,6 +4,11 @@
 
 ### Added
 
+- [#1798](https://github.com/epiphany-platform/epiphany/issues/1798) - Additional alerts for Prometheus
+- [#1355](https://github.com/epiphany-platform/epiphany/issues/1355) - Updating cloud based OS images - configuration required for Azure RHEL LVM images
+- [#2081](https://github.com/epiphany-platform/epiphany/issues/2081) - Replace Skopeo with Crane
+- [#1323](https://github.com/epiphany-platform/epiphany/issues/1323) - Documentation how to handle data in Opendistro for Elasticsearch
+
 ### Fixed
 
 - [#1870](https://github.com/epiphany-platform/epiphany/issues/1870) - Do not install Filebeat when there is no Elasticsearch
@@ -14,14 +19,25 @@
 - [#1979](https://github.com/epiphany-platform/epiphany/issues/1979) - RabbitMQ fails on upgrade when 2 nodes are specified that are not clustered
 - [#1984](https://github.com/epiphany-platform/epiphany/issues/1984) - RabbitMQ 3.7.10 fails on upgrade to 3.8.9: 'rabbitmqctl version' command not found
 - [#1824](https://github.com/epiphany-platform/epiphany/issues/1824) - offline installation fails with error 'yum lockfile is held by another process' (Azure/RHEL)
+- [#2069](https://github.com/epiphany-platform/epiphany/issues/2069) - [CentOS] epicli fails on task [repository : Create epirepo repository]
+- [#2066](https://github.com/epiphany-platform/epiphany/issues/2066) - [CentOS] download-requirements.sh fails on extracting tar with backed up repos
+- [#2067](https://github.com/epiphany-platform/epiphany/issues/2067) - [CentOS] epicli fails on task "repository : Wait for yum lock to be released" on CentOS Minimal
+- [#2115](https://github.com/epiphany-platform/epiphany/issues/2115) - Epicli hangs on importing GPG keys for kubernetes repository on RHEL
+- [#2121](https://github.com/epiphany-platform/epiphany/issues/2121) - [RedHat/CentOS] Erlang package versions specified in requirements are missing in external repository
+- [#2068](https://github.com/epiphany-platform/epiphany/issues/2068) - Preflight role requires sudoer user
 
 ### Updated
 
 - [#1953](https://github.com/epiphany-platform/epiphany/issues/1953) - Replace Pipenv with Poetry
 - [#1862](https://github.com/epiphany-platform/epiphany/issues/1862) - Upgrade Ignite (2.9.1)
 - [#1952](https://github.com/epiphany-platform/epiphany/issues/1952) - Upgrade ansible to 2.10.x
 - [#1864](https://github.com/epiphany-platform/epiphany/issues/1864) - Upgrade Hashicorp Vault (1.6.1), Vault Helm Chart (0.9.0), Vault-k8s (0.7.0)
+- [#2029](https://github.com/epiphany-platform/epiphany/issues/2029) - Remove old ARM references.
+- [#1901](https://github.com/epiphany-platform/epiphany/issues/1901) - Make Epiphany upgrades selective (Kafka). Added new parameter for epicli (--upgrade-components)
+- [#2080](https://github.com/epiphany-platform/epiphany/issues/2080) - Update RHEL images in CI pipelines and documentation to the current latest 7.9 version
+- [#1859](https://github.com/epiphany-platform/epiphany/issues/1859) - Upgrade Open Distro for Elasticsearch to v1.13.x and elasticsearch-oss to v7.10.2
 
 ### Breaking changes
+- Feature name: `elasticsearch` removed in favor of feature name: `opendistro-for-elasticsearch`.
 
 ### Known issues

core/src/epicli/.devcontainer/pyproject.toml

@@ -14,7 +14,6 @@ python-json-logger = "*"
 ansible = "==2.10.5"
 terraform-bin = "*"
 azure-cli = "==2.8.0"
-skopeo-bin = "*"
 "ruamel.yaml" = "*"
 
 [build-system]

core/src/epicli/.devcontainer/requirements.txt

@@ -141,7 +141,6 @@ ruamel.yaml==0.16.12
 s3transfer==0.3.4
 scp==0.13.3
 six==1.15.0; python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version < "4" and python_version >= "3.6"
-skopeo-bin==1.0.3
 sshtunnel==0.1.5
 tabulate==0.8.7
 terraform-bin==1.0.1

core/src/epicli/.vscode/launch.json

@@ -24,6 +24,7 @@
             // "args": ["test", "-b", "${workspaceFolder}/clusters/build/<DIR>"]
             // "args": ["test", "-b", "${workspaceFolder}/clusters/build/<DIR>", "-g", "<TEST_GROUP>"]
             // "args": ["upgrade", "-b", "${workspaceFolder}/clusters/build/<DIR>"]
+            // "args": ["upgrade", "-b", "${workspaceFolder}/clusters/build/<DIR>","--upgrade-components","kafka,ignite"]
         },
         {
             "name": "unit tests",

core/src/epicli/cli/engine/ApplyEngine.py

@@ -8,11 +8,13 @@
 from cli.helpers.build_saver import save_manifest, get_inventory_path
 from cli.helpers.yaml_helpers import safe_load_all
 from cli.helpers.Log import Log
+from cli.helpers.os_images import get_os_distro_normalized
 from cli.engine.providers.provider_class_loader import provider_class_loader
 from cli.engine.schema.DefaultMerger import DefaultMerger
 from cli.engine.schema.SchemaValidator import SchemaValidator
 from cli.engine.schema.ConfigurationAppender import ConfigurationAppender
 from cli.engine.terraform.TerraformTemplateGenerator import TerraformTemplateGenerator
+from cli.engine.terraform.TerraformFileCopier import TerraformFileCopier
 from cli.engine.terraform.TerraformRunner import TerraformRunner
 from cli.engine.ansible.AnsibleRunner import AnsibleRunner
 
@@ -122,39 +124,13 @@ def assert_consistent_os_family(self):
         # Before this issue https://github.com/epiphany-platform/epiphany/issues/195 gets resolved,
         # we are forced to do assertion here.
 
-        def _get_os_indicator(vm_doc):
-            expected_indicators = {
-                "ubuntu": "ubuntu",
-                "rhel": "redhat",
-                "redhat": "redhat",
-                "centos": "centos",
-            }
-            if vm_doc.provider == "azure":
-                # Example image offers:
-                # - UbuntuServer
-                # - RHEL
-                # - CentOS
-                for indicator in expected_indicators:
-                    if indicator in vm_doc.specification.storage_image_reference.offer.lower():
-                        return expected_indicators[indicator]
-            if vm_doc.provider == "aws":
-                # Example public/official AMI names:
-                # - ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200611
-                # - RHEL-7.8_HVM_GA-20200225-x86_64-1-Hourly2-GP2
-                # - CentOS 7.8.2003 x86_64
-                for indicator in expected_indicators:
-                    if indicator in vm_doc.specification.os_full_name.lower():
-                        return expected_indicators[indicator]
-            # When name is completely custom we just skip the check
-            return None
-
         virtual_machine_docs = select_all(
             self.infrastructure_docs,
             lambda x: x.kind == 'infrastructure/virtual-machine',
         )
 
         os_indicators = {
-            _get_os_indicator(vm_doc)
+            get_os_distro_normalized(vm_doc)
             for vm_doc in virtual_machine_docs
         }
 
@@ -177,6 +153,10 @@ def apply(self):
             with TerraformTemplateGenerator(self.cluster_model, self.infrastructure_docs) as template_generator:
                 template_generator.run()
 
+            # Copy cloud-config.yml since it contains bash code which can't be templated easily (requires {% raw %}...{% endraw %})
+            with TerraformFileCopier(self.cluster_model, self.infrastructure_docs) as file_copier:
+                file_copier.run()
+
             # Run Terraform to create infrastructure
             with TerraformRunner(self.cluster_model, self.configuration_docs) as tf_runner:
                 tf_runner.build()

core/src/epicli/cli/engine/PrepareEngine.py

@@ -28,7 +28,6 @@ def __exit__(self, exc_type, exc_value, traceback):
     def prepare(self):
         prepare_src = os.path.join(self.PREPARE_PATH, self.os)
         charts_src = self.CHARTS_PATH
-        skopeo_src = os.path.join(dirname(dirname(inspect.getfile(os))), 'skopeo_linux')
 
         prepare_dst = os.path.join(Config().output_dir, 'prepare_scripts')
         charts_dst = os.path.join(prepare_dst, 'charts', 'system')
@@ -37,16 +36,11 @@ def prepare(self):
             supported_os = os.listdir(self.PREPARE_PATH)
             raise Exception(f'Unsupported OS: {self.os}. Currently supported: {supported_os}')
 
-        if not os.path.exists(skopeo_src):
-            raise Exception('Skopeo dependency not found')
-
         # copy files to output dir
         copy_files_recursively(prepare_src, prepare_dst)
         copy_files_recursively(charts_src, charts_dst)
-        shutil.copy(skopeo_src, prepare_dst)
 
-        # make sure the scripts and skopeo are executable
-        self.make_file_executable(os.path.join(prepare_dst, 'skopeo_linux'))
+        # make sure the scripts are executable
         self.make_file_executable(os.path.join(prepare_dst, 'download-requirements.sh'))
 
         self.logger.info(f'Prepared files for downloading the offline requirements in: {prepare_dst}')

core/src/epicli/cli/engine/ansible/AnsibleRunner.py

@@ -54,10 +54,6 @@ def copy_resources(self):
         shutil.rmtree(ansible_dir, ignore_errors=True)
         copy_files_recursively(AnsibleRunner.ANSIBLE_PLAYBOOKS_PATH, ansible_dir)
 
-        # copy skopeo so Ansible can move it to the repositry machine
-        if not Config().offline_requirements:
-            shutil.copy(os.path.join(dirname(dirname(inspect.getfile(os))), 'skopeo_linux'), '/tmp')
-
     def pre_flight(self, inventory_path):
         self.logger.info('Checking connection to each machine')
         self.ansible_command.run_task_with_retries(inventory=inventory_path,

core/src/epicli/cli/engine/ansible/AnsibleVarsGenerator.py

@@ -133,6 +133,7 @@ def populate_group_vars(self, ansible_dir):
         main_vars['wait_for_pods'] = Config().wait_for_pods
         main_vars['is_upgrade_run'] = self.is_upgrade_run
         main_vars['roles_with_generated_vars'] = sorted(self.roles_with_generated_vars)
+        main_vars['upgrade_components'] = Config().upgrade_components
 
         if self.is_upgrade_run:
             shared_config_doc = self.get_shared_config_from_manifest()