Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE REQUEST][download-requirements] Optimize files downloading #3116

Closed
10 of 20 tasks
sbbroot opened this issue Apr 28, 2022 · 0 comments
Closed
10 of 20 tasks

[FEATURE REQUEST][download-requirements] Optimize files downloading #3116

sbbroot opened this issue Apr 28, 2022 · 0 comments
Assignees
@przemyslavic @sbbroot
Labels
area/development python Pull requests that update Python code

Comments

@sbbroot
Copy link
Contributor

sbbroot commented Apr 28, 2022 •
edited by seriva
Loading

Is your feature request related to a problem? Please describe.
Currently we download all files as is, but we would like to download only those needed by the user.

Describe the solution you'd like
Each file entry in the requirements file should have an additional key called deps, which will contain a list of components that use such file. Check should be made based on deps and info extracted from the manifest's roles mapping.

Describe alternatives you've considered
.

Additional context
image.png

DoD checklist

  • Changelog
    • updated
    • not needed
  • COMPONENTS.md
    • updated
    • not needed
  • Schema
    • updated
    • not needed
  • Backport tasks
    • created
    • not needed
  • Documentation
    • added
    • updated
    • not needed
  • Feature has automated tests
  • Automated tests passed (QA pipelines)
    • apply
    • upgrade
    • backup/restore
  • Idempotency tested
  • All conversations in PR resolved
  • Solution meets requirements and is done according to design doc
  • Usage compliant with license
@sbbroot sbbroot added area/development python Pull requests that update Python code status/grooming-needed labels Apr 28, 2022
@sbbroot sbbroot mentioned this issue Apr 28, 2022
Closed
5 tasks
@sbbroot sbbroot self-assigned this May 19, 2022
sbbroot added a commit to sbbroot/epiphany that referenced this issue May 20, 2022
@sbbroot
Optimize files downloading (hitachienergy#3116)
a4fe752
sbbroot added a commit to sbbroot/epiphany that referenced this issue May 20, 2022
@sbbroot
Optimize files downloading (hitachienergy#3116)
2cff77a
sbbroot added a commit to sbbroot/epiphany that referenced this issue May 20, 2022
@sbbroot
Optimize files downloading (hitachienergy#3116)
d371e62
sbbroot added a commit that referenced this issue May 23, 2022
@sbbroot
Optimize files downloading (#3116) (#3156)
96d9c41
sbbroot added a commit that referenced this issue Jun 8, 2022
@sbbroot
Optimize files downloading (#3116) (#3156)
4db1fc9
sbbroot added a commit that referenced this issue Jun 14, 2022
@sbbroot
Optimize files downloading (#3116) (#3156)
ceb2cca
sbbroot added a commit that referenced this issue Jun 28, 2022
@sbbroot
Optimize files downloading (#3116) (#3156)
73e6ad4
sbbroot added a commit that referenced this issue Jun 28, 2022
@sbbroot
[2.0.1] Adaptive mode for downloading requirements (#3188)
0229549 
* Split available_roles and roles_mapping into separate yaml documents (#3097) (#3119)

* available_roles splitted into feature-mappings and features
  documents

* feature-mappings added to the Init by default

* Add manifest file parsing (#3105) (#3130)

* Add `-m/--manifest` flag to accept manifest.yml produced by
  `epicli init/prepare`

* Add `-v/--verbose` mode for printing out parsed manifest data

* Add ManifestReader class used for paring the manifest.yml file

* Move src/command/*.py to debian/redhat subdirs where needed

* Optimize Grafana dashboards downloading (#3131) (#3150)

* Optimize files downloading (#3116) (#3156)

* Add image-registry configuration reading (#3106) (#3159)
@przemyslavic przemyslavic self-assigned this Jun 28, 2022
@seriva seriva closed this as completed Jun 29, 2022
seriva added a commit that referenced this issue Aug 9, 2022
@seriva @plirglo
@to-bar @sbbroot @przemyslavic @rafzei @cicharka @atsikham @tomasz-baran
Merge develop into 2.0 (#3250)
ad4c3d3 
* Document upgrade Red Hat / CentOS 7 to v 8.x (#3109)

* Migrate registry volume to named one (#3118)

* Fix dnf repoquery checks only latest kube* packages (#3123) (#3126)

* Switch to RHEL 8.4 for longer support (#3117) (#3129)

Co-authored-by: Tomasz Baran <[email protected]>

* [develop] Fetch missing packages, add stderr handling   (#3132)

* Add handling stderr to repoquery and check if there are missing packages

* Skip missing_packages if dependencies, handle output of dnf download

* Unit tests fix

* Changes after review, Update libmodulemd to the latest

* Documentation update

* Remove duplicated run(), fix for offline mode

* Tests fix

* Lifecycle update (#3135)

* Lifecycle update

* Mark 1.2 as out of support

* Bump epicli version in develop (#3134)

Co-authored-by: przemyslavic <>

* Add policycoreutils package (#3139)

* Add allow_mismatch flag for ceph/ceph:v16.2.7 image (#3136) (#3138)

* [2.0.1] Filebeat upgrade to 7.12.1 (#3086)

* Filebeat update to 7.12.1

* Add missing tasks name in upgrade/filebeat.yml

* Update sha256

* Update changelogs

* Changelog change back to 2.0.1

Co-authored-by: przemyslavic <[email protected]>

* [2.0.1] k8s-modules: update documentation (#3146)

Result of spike: #2982

Signed-off-by: cicharka <[email protected]>

* Support 'epicli upgrade' for RHEL/AlmaLinux 8 (#3154)

* Upgrade only to RHEL 8.4

* Disable legacy containerd plugin to avoid instance auto-recovery on AWS

* Reboot system after update only when needed

* Update Leapp metadata file

* Enable yum repos after OS is updated

* Use target option

* Handle PostgreSQL packages

* Enable upgrade mode for RedHat OS family

* Add releasever parameter

* Fix update of libmodulemd package

* Remove releasever DNF variable

* Suspend HealthCheck process on AWS

* Install python3-psycopg2 package also for RedHat family

* Add ntsysv package for Azure

* Prevent auto-upgrade of repmgr10-4.0.6-1.el7

* Update changelog

* [2.0.1] Migration to OpenSearch (#3093)

* All in one commit - from PR #2983

* Ansible-lint adjustments

* Remove leftovers

* Tests fix

* Adjust download-requirements

* HA fix, improvements

* Fix defaults, schema

* Fixes in migration to opensearch and opensearch dashboards, add cleanup

* Improvements

* Changes after review

* Update doc, changelog and schema after review

* Spec tests update, rebase + changes after review

* Fix defaults

* Fix unittest

* Fix backup/restore

* Replace kibana with opensearch_dashboards

* Fix apply mode, cleanup, add opensearch spec test

* Disable upgrade of logging/opensearch, cleanup and rename vars

* [2.0.1] Add ARM architecture support for AlmaLinux 8.4 (#3151)

* merge ARM installation

* Add repositories ids

* Modify SHAs

* Merge in develop changes

* Add policycoreutils to packages list

* Add docs and lua package

* Modification after review

* Fix Config.py for supported architectures (#3175)

* [2.0.1] Allow temporary credentials (session token parameter) (#3076)

* filebeat: update template for new version (#3141)

* Fix after bumping up filebeat version (PR: #3086).
* related to k8s_as_cloud_service flag

* Source containerd version and allow downgrade (#3170)

* [2.0.1] Bumped Python packages (#3176)

* Bumping Python packages.

* Added changelog

* Added Sonarcloud status badges. (#3182)

* [2.0.1] Add sssd and dependencies to requirements (#3155)

* Add ssd packages needed to upgrade ssd to v2.6.2

* [2.0.1] Low hanging fruit SonarQube fixes. (#3183)

* SonarQube fixes

* [2.0.1] Fix `use_network_security_groups` is set to `false` (#3181)

* Fix `use_network_security_groups` is set to `false`
* SonarQube fix.
* Minor fix after review

* Ensure ca-certificates package is in the latest version (#3169)

* Ensure ca-certificates package is in the latest version

* Add tar to base packages for RHEL mode

* Ensure tar is not uninstalled too early

* Use constants instead of string literals

* Ignore non-critical DNF error

* Ensure dnf config-manager command

* Do not use constants for better readability

* Ensure epel repo is enabled

* Fix is_repo_enabled method

* Preserve epel-release package

* Remove accidental import

* Apply suggestions from code review

* Apply suggestions from 2nd review

* Fix `The same or higher version of epel-release is already installed` error

* Create a YAML build pipeline (#3187)

* Fix PostgreSQL tests (#3192)

* Fix postgresql tests

* Update default configurations

* Restore escaping for PostgreSQL tests (#3195)

* Ensure epicli upgrade works on cluster with upgraded RHEL from version 7 to 8 (#3191)

* Fix repmgr10 service

* Fix for K8s master with Calico

* Mark AWS instances as healthy

* Suspend ReplaceUnhealthy process

* Put all instances into Standby state and disable auto-recovery

* Keep ReplaceUnhealthy process suspended

* Remove unsupported Pylint options (#3197)

* Add ARM dependencies (#3185)

* Add ARM dependencies

* Add rook to unsupported roles

* Add FELIX_IPTABLESBACKEND variable to calico configuration

* Update documentation

* Add FELIX_IPTABLESBACKEND
for ARM only

* Remove rook from requirements

* Changelog: move ARM to 2.0.1

* [2.0.1] Adaptive mode for downloading requirements (#3188)

* Split available_roles and roles_mapping into separate yaml documents (#3097) (#3119)

* available_roles splitted into feature-mappings and features
  documents

* feature-mappings added to the Init by default

* Add manifest file parsing (#3105) (#3130)

* Add `-m/--manifest` flag to accept manifest.yml produced by
  `epicli init/prepare`

* Add `-v/--verbose` mode for printing out parsed manifest data

* Add ManifestReader class used for paring the manifest.yml file

* Move src/command/*.py to debian/redhat subdirs where needed

* Optimize Grafana dashboards downloading (#3131) (#3150)

* Optimize files downloading (#3116) (#3156)

* Add image-registry configuration reading (#3106) (#3159)

* Fix ansible-lint scan location (#3203)

* Fix ansible-lint scan location

* Update ansible_lint_error_threshold

* Allow excluding test groups (#3202)

* Add excluding test groups

* Exclude effective test groups

* Update doc

* Update configuration

* Do not use sets

* Apply suggestions from review

* Fix failed services after RHEL 7 upgrade on cluster created with epicli v1.3 (#3204)

* Fix failed services

* Fix for offline mode

* Workaround for esl-erlang package issue (#3211)

* Provide kubeconfig file for spec tests (#3206)

* Comply with Rubocop

* Print selected groups as yaml

* Provide kubeconfig file for spec tests

* Fix Pylint import-error issues in VSCode

* Self code review

* Add new option to launch configurations

* Fix after tests

* Update rubocop_linter_threshold

* Apply suggestions from code review

* Print selected test groups before preparing env

* Added checking enabled roles for feature. (#3213)

* ceph: fix tag for ceph image (#3199)

* Use a stable tag for the quay.io/ceph/ceph:v16.2.7 image

Signed-off-by: cicharka <[email protected]>

* Added ability to disable OpenSearch audit logs (#3215)

* Added ability to disable OpenSearch audit logs

* Added Black Duck Scan plugin (#3219)

* Added Black Duck Scan plugin

* Add java to devcontainer to run BDS.

* Change to JAVA headless

* Cache is already cleaned, removed unneeded run.

* Support for original output coloring (#3220)

* Add click package

* Support for original output coloring

* Add click package to CI pipeline

* Fix CI task

* Use human friendly color codes

* Fix naming style

* Do not detect log level for colored loggers

* Apply --no-color option for epicli output formatter

* Highlight info on Ansible commands

* Fix UncolorJsonFormatter

* Update changelog

* Update pylint_score_cli_threshold

* Better formatting

* Add support for NO_COLOR env var

* Use 'python3 -m pip' instead of pip

* Ensure click

* Restore higher threshold

* Use python3 -m pylint

* Fix pylint_score_cli_threshold

* Resolve dependencies for specified package version (#3223)

* Fix issues reported by Pylint

* Resolve dependencies for specified version

* Update changelog

* Update crane to v0.11.0 (#3230)

* Fix disabling rook in feature-mappings (#3227)

* Run rook playbook on rook group

* Move rook images under rook group

* Update LIFECYCLE.md (#3235)

* Update LIFECYCLE.md

* Skip firewall role unless present in inventory (#3233)

* Update Calico and Canal to fix issue on ARM (#3228)

* Update Calico and Canal to fix issue on ARM

* Use single arch for CNI plugin images

* Fix incorrect checksums

* OpenSearch: add dedicated user for Filebeat (#3079) (#3221)

* removes previously used `logstash` user from filebeat configuration
* removes `logstash` user from demo users configured by opensearch
* enables creation of dedicated filebeat user - by default name set to
  `filebeatservice`
* add user detection in case of re-apply
* set user `kibanaserver` and `filebeatservice` installation
  dependent on inventory groups rather than user_active
  flag (previously configured by users)
* simplify documentation
* set dashboards hosts list based on their group
* use yaml anchors in user manipulation tasks

* Add filtering mechanism for the sensitive data (#3207) (#3208)

* Add filtering mechanism for the sensitive data (#3207)

* Include aws-cli and git in Dockerfile #2982 (#3236)

* Include aws-cli and git in Dockerfile
* add components entries

Signed-off-by: cicharka <[email protected]>

* Fix getting package dependencies (#3239)

* Optimize get_package_dependencies method

* Update changelog

* Simplify method in CommandRunMock class

* Add test_get_package_dependencies_return_value

* Apply suggestions from code review

* Sort imports

* Move APT_CACHE_DEPENDS_DATA over test

* Fix k8s_as_cloud_service flag used in download-requirements (#3222) (#3242)

* Fix k8s_as_cloud_service flag used in download-requirements (#3222)

* filebeat: fix template for k8s_as_cloud_service (#3247)

Signed-off-by: cicharka <[email protected]>

* k8s: controller managed attachment and detachment #3190 (#3237)

* enable configuration of enable-controller-attach-detach
  kubelet parameter in input manifest
* set enable-controller-attach-detach to true
* fix extend-kubeadm-config.yml task in order to keep
  consistent values in configMaps and kubeadm-config.yml
* move get and set cluster version utils to kubernetes_common
* Update docs/home/howto/kubernetes/PERSISTENT_STORAGE.md

* Remove leftovers of OpenDistro repository (#3248)

* Add haproxy to k8s images group (#3240)

* fix for #3231
* enhance test data for image_requirements
* add type hints

* Fix handling of download-requirements flag file (#3246)

Co-authored-by: Irek Głownia <[email protected]>
Co-authored-by: Tomasz Baran <[email protected]>
Co-authored-by: sbbroot <[email protected]>
Co-authored-by: przemyslavic <[email protected]>
Co-authored-by: Rafal Zeidler <[email protected]>
Co-authored-by: cicharka <[email protected]>
Co-authored-by: Anatoli Tsikhamirau <[email protected]>
Co-authored-by: Tomasz Baran <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@przemyslavic przemyslavic

@sbbroot sbbroot

Labels
area/development python Pull requests that update Python code
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@seriva @przemyslavic @sbbroot