Merge develop into 2.0 (#3250)

* Document upgrade Red Hat / CentOS 7 to v 8.x (#3109)

* Migrate registry volume to named one (#3118)

* Fix dnf repoquery checks only latest kube* packages (#3123) (#3126)

* Switch to RHEL 8.4 for longer support (#3117) (#3129)

Co-authored-by: Tomasz Baran <[email protected]>

* [develop] Fetch missing packages, add stderr handling   (#3132)

* Add handling stderr to repoquery and check if there are missing packages

* Skip missing_packages if dependencies, handle output of dnf download

* Unit tests fix

* Changes after review, Update libmodulemd to the latest

* Documentation update

* Remove duplicated run(), fix for offline mode

* Tests fix

* Lifecycle update (#3135)

* Lifecycle update

* Mark 1.2 as out of support

* Bump epicli version in develop (#3134)

Co-authored-by: przemyslavic <>

* Add policycoreutils package (#3139)

* Add allow_mismatch flag for ceph/ceph:v16.2.7 image (#3136) (#3138)

* [2.0.1] Filebeat upgrade to 7.12.1 (#3086)

* Filebeat update to 7.12.1

* Add missing tasks name in upgrade/filebeat.yml

* Update sha256

* Update changelogs

* Changelog change back to 2.0.1

Co-authored-by: przemyslavic <[email protected]>

* [2.0.1] k8s-modules: update documentation (#3146)

Result of spike: #2982

Signed-off-by: cicharka <[email protected]>

* Support 'epicli upgrade' for RHEL/AlmaLinux 8 (#3154)

* Upgrade only to RHEL 8.4

* Disable legacy containerd plugin to avoid instance auto-recovery on AWS

* Reboot system after update only when needed

* Update Leapp metadata file

* Enable yum repos after OS is updated

* Use target option

* Handle PostgreSQL packages

* Enable upgrade mode for RedHat OS family

* Add releasever parameter

* Fix update of libmodulemd package

* Remove releasever DNF variable

* Suspend HealthCheck process on AWS

* Install python3-psycopg2 package also for RedHat family

* Add ntsysv package for Azure

* Prevent auto-upgrade of repmgr10-4.0.6-1.el7

* Update changelog

* [2.0.1] Migration to OpenSearch (#3093)

* All in one commit - from PR #2983

* Ansible-lint adjustments

* Remove leftovers

* Tests fix

* Adjust download-requirements

* HA fix, improvements

* Fix defaults, schema

* Fixes in migration to opensearch and opensearch dashboards, add cleanup

* Improvements

* Changes after review

* Update doc, changelog and schema after review

* Spec tests update, rebase + changes after review

* Fix defaults

* Fix unittest

* Fix backup/restore

* Replace kibana with opensearch_dashboards

* Fix apply mode, cleanup, add opensearch spec test

* Disable upgrade of logging/opensearch, cleanup and rename vars

* [2.0.1] Add ARM architecture support for AlmaLinux 8.4 (#3151)

* merge ARM installation

* Add repositories ids

* Modify SHAs

* Merge in develop changes

* Add policycoreutils to packages list

* Add docs and lua package

* Modification after review

* Fix Config.py for supported architectures (#3175)

* [2.0.1] Allow temporary credentials (session token parameter) (#3076)

* filebeat: update template for new version (#3141)

* Fix after bumping up filebeat version (PR: #3086).
* related to k8s_as_cloud_service flag

* Source containerd version and allow downgrade (#3170)

* [2.0.1] Bumped Python packages (#3176)

* Bumping Python packages.

* Added changelog

* Added Sonarcloud status badges. (#3182)

* [2.0.1] Add sssd and dependencies to requirements (#3155)

* Add ssd packages needed to upgrade ssd to v2.6.2

* [2.0.1] Low hanging fruit SonarQube fixes. (#3183)

* SonarQube fixes

* [2.0.1] Fix `use_network_security_groups` is set to `false` (#3181)

* Fix `use_network_security_groups` is set to `false`
* SonarQube fix.
* Minor fix after review

* Ensure ca-certificates package is in the latest version (#3169)

* Ensure ca-certificates package is in the latest version

* Add tar to base packages for RHEL mode

* Ensure tar is not uninstalled too early

* Use constants instead of string literals

* Ignore non-critical DNF error

* Ensure dnf config-manager command

* Do not use constants for better readability

* Ensure epel repo is enabled

* Fix is_repo_enabled method

* Preserve epel-release package

* Remove accidental import

* Apply suggestions from code review

* Apply suggestions from 2nd review

* Fix `The same or higher version of epel-release is already installed` error

* Create a YAML build pipeline (#3187)

* Fix PostgreSQL tests (#3192)

* Fix postgresql tests

* Update default configurations

* Restore escaping for PostgreSQL tests (#3195)

* Ensure epicli upgrade works on cluster with upgraded RHEL from version 7 to 8 (#3191)

* Fix repmgr10 service

* Fix for K8s master with Calico

* Mark AWS instances as healthy

* Suspend ReplaceUnhealthy process

* Put all instances into Standby state and disable auto-recovery

* Keep ReplaceUnhealthy process suspended

* Remove unsupported Pylint options (#3197)

* Add ARM dependencies (#3185)

* Add ARM dependencies

* Add rook to unsupported roles

* Add FELIX_IPTABLESBACKEND variable to calico configuration

* Update documentation

* Add FELIX_IPTABLESBACKEND
for ARM only

* Remove rook from requirements

* Changelog: move ARM to 2.0.1

* [2.0.1] Adaptive mode for downloading requirements (#3188)

* Split available_roles and roles_mapping into separate yaml documents (#3097) (#3119)

* available_roles splitted into feature-mappings and features
  documents

* feature-mappings added to the Init by default

* Add manifest file parsing (#3105) (#3130)

* Add `-m/--manifest` flag to accept manifest.yml produced by
  `epicli init/prepare`

* Add `-v/--verbose` mode for printing out parsed manifest data

* Add ManifestReader class used for paring the manifest.yml file

* Move src/command/*.py to debian/redhat subdirs where needed

* Optimize Grafana dashboards downloading (#3131) (#3150)

* Optimize files downloading (#3116) (#3156)

* Add image-registry configuration reading (#3106) (#3159)

* Fix ansible-lint scan location (#3203)

* Fix ansible-lint scan location

* Update ansible_lint_error_threshold

* Allow excluding test groups (#3202)

* Add excluding test groups

* Exclude effective test groups

* Update doc

* Update configuration

* Do not use sets

* Apply suggestions from review

* Fix failed services after RHEL 7 upgrade on cluster created with epicli v1.3 (#3204)

* Fix failed services

* Fix for offline mode

* Workaround for esl-erlang package issue (#3211)

* Provide kubeconfig file for spec tests (#3206)

* Comply with Rubocop

* Print selected groups as yaml

* Provide kubeconfig file for spec tests

* Fix Pylint import-error issues in VSCode

* Self code review

* Add new option to launch configurations

* Fix after tests

* Update rubocop_linter_threshold

* Apply suggestions from code review

* Print selected test groups before preparing env

* Added checking enabled roles for feature. (#3213)

* ceph: fix tag for ceph image (#3199)

* Use a stable tag for the quay.io/ceph/ceph:v16.2.7 image

Signed-off-by: cicharka <[email protected]>

* Added ability to disable OpenSearch audit logs (#3215)

* Added ability to disable OpenSearch audit logs

* Added Black Duck Scan plugin (#3219)

* Added Black Duck Scan plugin

* Add java to devcontainer to run BDS.

* Change to JAVA headless

* Cache is already cleaned, removed unneeded run.

* Support for original output coloring (#3220)

* Add click package

* Support for original output coloring

* Add click package to CI pipeline

* Fix CI task

* Use human friendly color codes

* Fix naming style

* Do not detect log level for colored loggers

* Apply --no-color option for epicli output formatter

* Highlight info on Ansible commands

* Fix UncolorJsonFormatter

* Update changelog

* Update pylint_score_cli_threshold

* Better formatting

* Add support for NO_COLOR env var

* Use 'python3 -m pip' instead of pip

* Ensure click

* Restore higher threshold

* Use python3 -m pylint

* Fix pylint_score_cli_threshold

* Resolve dependencies for specified package version (#3223)

* Fix issues reported by Pylint

* Resolve dependencies for specified version

* Update changelog

* Update crane to v0.11.0 (#3230)

* Fix disabling rook in feature-mappings (#3227)

* Run rook playbook on rook group

* Move rook images under rook group

* Update LIFECYCLE.md (#3235)

* Update LIFECYCLE.md

* Skip firewall role unless present in inventory (#3233)

* Update Calico and Canal to fix issue on ARM (#3228)

* Update Calico and Canal to fix issue on ARM

* Use single arch for CNI plugin images

* Fix incorrect checksums

* OpenSearch: add dedicated user for Filebeat (#3079) (#3221)

* removes previously used `logstash` user from filebeat configuration
* removes `logstash` user from demo users configured by opensearch
* enables creation of dedicated filebeat user - by default name set to
  `filebeatservice`
* add user detection in case of re-apply
* set user `kibanaserver` and `filebeatservice` installation
  dependent on inventory groups rather than user_active
  flag (previously configured by users)
* simplify documentation
* set dashboards hosts list based on their group
* use yaml anchors in user manipulation tasks

* Add filtering mechanism for the sensitive data (#3207) (#3208)

* Add filtering mechanism for the sensitive data (#3207)

* Include aws-cli and git in Dockerfile #2982 (#3236)

* Include aws-cli and git in Dockerfile
* add components entries

Signed-off-by: cicharka <[email protected]>

* Fix getting package dependencies (#3239)

* Optimize get_package_dependencies method

* Update changelog

* Simplify method in CommandRunMock class

* Add test_get_package_dependencies_return_value

* Apply suggestions from code review

* Sort imports

* Move APT_CACHE_DEPENDS_DATA over test

* Fix k8s_as_cloud_service flag used in download-requirements (#3222) (#3242)

* Fix k8s_as_cloud_service flag used in download-requirements (#3222)

* filebeat: fix template for k8s_as_cloud_service (#3247)

Signed-off-by: cicharka <[email protected]>

* k8s: controller managed attachment and detachment #3190 (#3237)

* enable configuration of enable-controller-attach-detach
  kubelet parameter in input manifest
* set enable-controller-attach-detach to true
* fix extend-kubeadm-config.yml task in order to keep
  consistent values in configMaps and kubeadm-config.yml
* move get and set cluster version utils to kubernetes_common
* Update docs/home/howto/kubernetes/PERSISTENT_STORAGE.md

* Remove leftovers of OpenDistro repository (#3248)

* Add haproxy to k8s images group (#3240)

* fix for #3231
* enhance test data for image_requirements
* add type hints

* Fix handling of download-requirements flag file (#3246)

Co-authored-by: Irek Głownia <[email protected]>
Co-authored-by: Tomasz Baran <[email protected]>
Co-authored-by: sbbroot <[email protected]>
Co-authored-by: przemyslavic <[email protected]>
Co-authored-by: Rafal Zeidler <[email protected]>
Co-authored-by: cicharka <[email protected]>
Co-authored-by: Anatoli Tsikhamirau <[email protected]>
Co-authored-by: Tomasz Baran <[email protected]>

.ansible-lint

@@ -38,6 +38,7 @@ skip_list:
   - meta-no-info
   - package-latest
   - fqcn-builtins
+  - no-jinja-when
 
 ##################
 # Tags to follow #

.devcontainer/Dockerfile

@@ -4,6 +4,7 @@ ARG USERNAME=vscode
 ARG USER_UID=1000
 ARG USER_GID=$USER_UID
 
+ARG AWS_CLI_VERSION=2.0.30
 ARG HELM_VERSION=3.3.1
 ARG KUBECTL_VERSION=1.22.4
 ARG TERRAFORM_VERSION=1.1.3
@@ -19,6 +20,7 @@ RUN : INSTALL APT REQUIREMENTS \
     make musl-dev openssh-client procps \
     psmisc rsync ruby-full sudo tar \
     unzip vim \
+    openjdk-11-jdk-headless \
  && apt-get -q autoremove -y \
  && apt-get -q clean -y \
  && rm -rf /var/lib/apt/lists/*
@@ -37,7 +39,13 @@ RUN : INSTALL HELM BINARY \
  && curl -fsSLO https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip \
  && unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin \
  && rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip \
- && terraform version
+ && terraform version \
+ && : INSTALL AWS CLI BINARY \
+ && curl -fsSLO https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip \
+ && unzip awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip \
+ && ./aws/install -i /usr/local/aws-cli -b /usr/local/bin \
+ && rm -rf awscli-exe-linux-x86_64-${AWS_CLI_VERSION}.zip ./aws \
+ && aws --version
 
 RUN : INSTALL GEM REQUIREMENTS \
  && gem install \
@@ -58,5 +66,8 @@ RUN : SETUP USER AND OTHERS \
  && chmod ug=r,o= /etc/sudoers.d/$USERNAME \
  && setcap 'cap_net_bind_service=+ep' /usr/bin/ssh
 
+RUN : SETUP JAVE_HOME
+ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/
+
 RUN : SETUP EPICLI ALIAS \
  && echo alias epicli='"export PYTHONPATH=/workspaces/epiphany && python3 -m cli.epicli"' >> /etc/bash.bashrc

.devcontainer/devcontainer.json

@@ -4,6 +4,8 @@
     "extensions": [
         // Ansible
         "redhat.ansible",
+        // Black Duck Scan
+        "synopsyscodesight.vscode-codesight",
         // Common
         "shardulm94.trailing-spaces",
         // Git