Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install xsaim packs with and without dataset #30301

Merged
merged 49 commits into from
Oct 31, 2023

Conversation

daryakoval
Copy link
Contributor

@daryakoval daryakoval commented Oct 18, 2023

Status

  • In Progress
  • Ready
  • In Hold - (Reason for hold)

Related Issues

fixes: https://jira-dc.paloaltonetworks.com/browse/CIAC-6529

Description

Different validation runs on pack installations when datasets exist on tenant and when not. Therefore we need to check the installation with the dataset on the tenant and without.

The flow in xsaim install packs should be:

  1. Clean XSIAM machine (it already exists no changes should be done).
  2. Delete dataset (POST '<base_url>/public_api/v1/xql/delete_dataset/')
  3. Install pack (check installation on the clear tenant). - already exists no changes should be done.
  4. Ingest dataset + Test modeling rules - already exists no changes should be done.
  5. (Only if test data exist) Uninstall the pack + Install the pack (Reinstall/Update) (In this step we need to check if all validation runs if installing the same version, or if the server ignores the installation if the pack is already installed with the same version).

My changes:

  • Added delete dataset to the Clean machines step.
  • Added Uninstall the pack + Install the pack check.
  • Please ignore the changes in Packs/qualys/ModelingRules/QualysModelingRules/ file. I will revert them before merging.

Must have

  • Tests
  • Documentation

Builds

Nightly that deletes datasets: https://code.pan.run/xsoar/content/-/jobs/35192465
Branch build that fails as expected: https://code.pan.run/xsoar/content/-/jobs/35464354
Nightly:

@daryakoval
Copy link
Contributor Author

Please ignore the changes in Packs/qualys/ModelingRules/QualysModelingRules/ file. I will revert them before merging.

@daryakoval daryakoval marked this pull request as ready for review October 29, 2023 14:10
Copy link
Contributor

@dantavori dantavori left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

amazing job :)
my notes are mainly about readability. you are adding something niche to central parts of the build - please go over your changes and make sure it's clear enough for someone who reads it for the first time without context.

@@ -136,6 +136,7 @@ validate-content-conf:
policy: pull-push
variables:
KUBERNETES_CPU_REQUEST: 2000m
EXTRACT_PRIVATE_TESTDATA: "true"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why if it's already in test_content_on_cloud_server_instances_base?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need it in the test collection step. I am collecting packs to reinstall, only if they have test data.

Tests/Marketplace/reinstall_packs.py Show resolved Hide resolved
Tests/Marketplace/reinstall_packs.py Show resolved Hide resolved
Tests/Marketplace/common.py Show resolved Hide resolved
Tests/Marketplace/search_and_uninstall_pack.py Outdated Show resolved Hide resolved
Tests/Marketplace/search_and_uninstall_pack.py Outdated Show resolved Hide resolved
Tests/scripts/collect_tests/collect_tests.py Outdated Show resolved Hide resolved
Copy link
Contributor

@dantavori dantavori left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noice!
see notes below and, of course, when ready revert the qualys pack changes.

Tests/Marketplace/reinstall_packs.py Show resolved Hide resolved
Tests/Marketplace/search_and_uninstall_pack.py Outdated Show resolved Hide resolved
@daryakoval daryakoval merged commit 771ca32 into master Oct 31, 2023
13 checks passed
@daryakoval daryakoval deleted the delete-dataset-installation branch October 31, 2023 07:18
sharonfi99 pushed a commit that referenced this pull request Oct 31, 2023
* adding delete dataset call for cleaning; changing Qualys for testing; adding reinstall packs list to collect tests;

* fix pre-commit

* auth id should be str

* adding logs

* adding global varible to download conf

* adding more logging

* delete logs

* add reinstall script

* fix path

* test without un-installation

* test without un-installation

* uninstallatio is mandatory

* adding got here logging

* adding more logs

* remove some logs to not wait

* save more logs, ucomment

* fixes to trigger nightly

* revert unnessasary changes

* delete logs

* testing

* revert changes

* ruining modeling rules before installation

* save second check

* check nightly

* revert changes

* save changed from cr

* save changed from cr

* revert qualys changes

* pre-commit fix
sapirshuker pushed a commit that referenced this pull request Dec 21, 2023
* adding delete dataset call for cleaning; changing Qualys for testing; adding reinstall packs list to collect tests;

* fix pre-commit

* auth id should be str

* adding logs

* adding global varible to download conf

* adding more logging

* delete logs

* add reinstall script

* fix path

* test without un-installation

* test without un-installation

* uninstallatio is mandatory

* adding got here logging

* adding more logs

* remove some logs to not wait

* save more logs, ucomment

* fixes to trigger nightly

* revert unnessasary changes

* delete logs

* testing

* revert changes

* ruining modeling rules before installation

* save second check

* check nightly

* revert changes

* save changed from cr

* save changed from cr

* revert qualys changes

* pre-commit fix
israelpoli pushed a commit that referenced this pull request May 15, 2024
* adding delete dataset call for cleaning; changing Qualys for testing; adding reinstall packs list to collect tests;

* fix pre-commit

* auth id should be str

* adding logs

* adding global varible to download conf

* adding more logging

* delete logs

* add reinstall script

* fix path

* test without un-installation

* test without un-installation

* uninstallatio is mandatory

* adding got here logging

* adding more logs

* remove some logs to not wait

* save more logs, ucomment

* fixes to trigger nightly

* revert unnessasary changes

* delete logs

* testing

* revert changes

* ruining modeling rules before installation

* save second check

* check nightly

* revert changes

* save changed from cr

* save changed from cr

* revert qualys changes

* pre-commit fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants