HC optimization finalization - phase 3 (#723) · IBM/network-config-analyzer@0c27129

Commit

HC optimization finalization - phase 3 (#723)

* Initial implementation of building and minimizing fw-rules directly from connectivity properties.

Signed-off-by: Tanya <[email protected]>

* Fixed lint errors.

Signed-off-by: Tanya <[email protected]>

* Updating (some of) expected results for explainability queries, according to more condensed optimized output.

Signed-off-by: Tanya <[email protected]>

* Fixed converting fw-rules to connectivity properties, while taking into account TCP/non-TCP protocol restriction.

Signed-off-by: Tanya <[email protected]>

* Optimized handling IpBlocks in optimized fw-rules minimization

Signed-off-by: Tanya <[email protected]>

* Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping.

Signed-off-by: Tanya <[email protected]>

* Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping.

Signed-off-by: Tanya <[email protected]>

* More optimization in calculation partial ns grouping.

Signed-off-by: Tanya <[email protected]>

* Fixed lint error

Signed-off-by: Tanya <[email protected]>

* Refining basic namespace grouping by finding more opportunities to use properties in containing connections.

Signed-off-by: Tanya <[email protected]>

* One more refinemenet of basic namespace grouping

Signed-off-by: Tanya <[email protected]>

* One more refinemenet of basic namespace grouping

Signed-off-by: Tanya <[email protected]>

* More refinemenets of peer grouping from properties

Signed-off-by: Tanya <[email protected]>

* More refinemenets of peer grouping from properties

Signed-off-by: Tanya <[email protected]>

* More refinements of peer grouping from properties

Signed-off-by: Tanya <[email protected]>

* Added outputEndpoints option handling to PeerSetElement.
Refined ns-set pairs grouping computation -trying starting from src_peers and from dst_peers and choosing a more compact grouping.
Added grouping by full IpBlock.

Signed-off-by: Tanya <[email protected]>

* Fixing lint errors.

Signed-off-by: Tanya <[email protected]>

* Fixing handling txt-no_fw_rules format in the optimized solution

Signed-off-by: Tanya <[email protected]>

* Fixing lint error

Signed-off-by: Tanya <[email protected]>

* Fix: taking into account connectivity restriction (TCP/non-TCP) in generation of dot output in optimized solution

Signed-off-by: Tanya <[email protected]>

* Small fixes in txt_no_fw_rules_format

Signed-off-by: Tanya <[email protected]>

* Small fixes in txt_no_fw_rules_format

Signed-off-by: Tanya <[email protected]>

* Added grouping by dns entries to the optimized algorithm.

Signed-off-by: Tanya <[email protected]>

* Changed expected results of connectivity map query tests according to optimized runs

Signed-off-by: Tanya <[email protected]>

* Changed optimized semantic diff query implementation according to the optimized fw-rules minimization algorithm.
Fixed get_connection_set_and_peers_from_cube.
Changed some of the expected results of semantic diff tests.

Signed-off-by: Tanya <[email protected]>

* Keeping every dns entry separate in minimization of fw rules.
Updated more semantic diff expected results.

Signed-off-by: Tanya <[email protected]>

* Updated containment, permits, forbids expected results.

Signed-off-by: Tanya <[email protected]>

* Cleaning up unused code and refactoring accordingly.

Signed-off-by: Tanya <[email protected]>

* Fixed lint error.

Signed-off-by: Tanya <[email protected]>

* Changed default to be the optimized run.

Signed-off-by: Tanya <[email protected]>

* In opt='debug' the result explanation should ne according to the optimized run.

Signed-off-by: Tanya <[email protected]>

* Restoring resource in scheme, changed by mistake.

Signed-off-by: Tanya <[email protected]>

* Updating more expected results.

Signed-off-by: Tanya <[email protected]>

* Small optimizations.

Signed-off-by: Tanya <[email protected]>

* Initial implementation of building and minimizing fw-rules directly from connectivity properties.

Signed-off-by: Tanya <[email protected]>

* Fixed lint errors.

Signed-off-by: Tanya <[email protected]>

* Updating (some of) expected results for explainability queries, according to more condensed optimized output.

Signed-off-by: Tanya <[email protected]>

* Fixed converting fw-rules to connectivity properties, while taking into account TCP/non-TCP protocol restriction.

Signed-off-by: Tanya <[email protected]>

* Optimized handling IpBlocks in optimized fw-rules minimization

Signed-off-by: Tanya <[email protected]>

* Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping.

Signed-off-by: Tanya <[email protected]>

* Optimized initial namespace grouping (by grouping few namespaces together, according to grouping in cubes). Also, added grouping by labels to initial grouping.

Signed-off-by: Tanya <[email protected]>

* More optimization in calculation partial ns grouping.

Signed-off-by: Tanya <[email protected]>

* Fixed lint error

Signed-off-by: Tanya <[email protected]>

* Refining basic namespace grouping by finding more opportunities to use properties in containing connections.

Signed-off-by: Tanya <[email protected]>

* One more refinemenet of basic namespace grouping

Signed-off-by: Tanya <[email protected]>

* One more refinemenet of basic namespace grouping

Signed-off-by: Tanya <[email protected]>

* More refinemenets of peer grouping from properties

Signed-off-by: Tanya <[email protected]>

* More refinemenets of peer grouping from properties

Signed-off-by: Tanya <[email protected]>

* More refinements of peer grouping from properties

Signed-off-by: Tanya <[email protected]>

* Added outputEndpoints option handling to PeerSetElement.
Refined ns-set pairs grouping computation -trying starting from src_peers and from dst_peers and choosing a more compact grouping.
Added grouping by full IpBlock.

Signed-off-by: Tanya <[email protected]>

* Fixing lint errors.

Signed-off-by: Tanya <[email protected]>

* Fixing handling txt-no_fw_rules format in the optimized solution

Signed-off-by: Tanya <[email protected]>

* Fixing lint error

Signed-off-by: Tanya <[email protected]>

* Fix: taking into account connectivity restriction (TCP/non-TCP) in generation of dot output in optimized solution

Signed-off-by: Tanya <[email protected]>

* Small fixes in txt_no_fw_rules_format

Signed-off-by: Tanya <[email protected]>

* Small fixes in txt_no_fw_rules_format

Signed-off-by: Tanya <[email protected]>

* Added grouping by dns entries to the optimized algorithm.

Signed-off-by: Tanya <[email protected]>

* Changed expected results of connectivity map query tests according to optimized runs

Signed-off-by: Tanya <[email protected]>

* Changed optimized semantic diff query implementation according to the optimized fw-rules minimization algorithm.
Fixed get_connection_set_and_peers_from_cube.
Changed some of the expected results of semantic diff tests.

Signed-off-by: Tanya <[email protected]>

* Keeping every dns entry separate in minimization of fw rules.
Updated more semantic diff expected results.

Signed-off-by: Tanya <[email protected]>

* Updated containment, permits, forbids expected results.

Signed-off-by: Tanya <[email protected]>

* Cleaning up unused code and refactoring accordingly.

Signed-off-by: Tanya <[email protected]>

* Fixed lint error.

Signed-off-by: Tanya <[email protected]>

* Changed default to be the optimized run.

Signed-off-by: Tanya <[email protected]>

* In opt='debug' the result explanation should ne according to the optimized run.

Signed-off-by: Tanya <[email protected]>

* Restoring resource in scheme, changed by mistake.

Signed-off-by: Tanya <[email protected]>

* Updating more expected results.

Signed-off-by: Tanya <[email protected]>

* Small optimizations.

Signed-off-by: Tanya <[email protected]>

* Small optimizations.

Signed-off-by: Tanya <[email protected]>

* Removed disjoint_ip_blocks from optimized solution

Signed-off-by: Tanya <[email protected]>

* Optimization: adding auto-connections to covered_peer_props (in fw-rules minimization) only if the number of peers is not too high, to avoid long run of these auto-connections calculation.

Signed-off-by: Tanya <[email protected]>

* Optimization: adding resources to global resource/namespace/pod list in test schemes (instead of putting them only in configurations), to avoid trying to load resources from live cluster, which is time-consuming.

Signed-off-by: Tanya <[email protected]>

* Deleted unused original implementation code.
Changed number of expected warnings (see Issue #724

Signed-off-by: Tanya <[email protected]>

* Further refining code by deleting unused original implementatation code.

Signed-off-by: Tanya <[email protected]>

* Removed usage of ConnectionSet.

Signed-off-by: Tanya <[email protected]>

* Fixed lint errors.
Fixed small error.

Signed-off-by: Tanya <[email protected]>

* Fixed sorting ConnectivityProperties (making stable sort)
Consequently, fixed expected results

Signed-off-by: Tanya <[email protected]>

* Fixed handling TCP / non-TCP connections output.

Signed-off-by: Tanya <[email protected]>

* More fixed expected results.

Signed-off-by: Tanya <[email protected]>

* More fixed expected results.
Small fix in printing TCP connections in dot format.

Signed-off-by: Tanya <[email protected]>

* Added some doc

Signed-off-by: Tanya <[email protected]>

* Removed no longer used named_ports and excluded_named_ports in ConnectivityProperties.
Removed outdated unit tests.

Signed-off-by: Tanya <[email protected]>

* Small fixes

Signed-off-by: Tanya <[email protected]>

* Updates of some more expected results.
Removed --optimized_run flag.
Removed ConnectionSet class.

Signed-off-by: Tanya <[email protected]>

* Fixed lint errors.
Removed original-to-optimized-comparison workflows from test-push.

Signed-off-by: Tanya <[email protected]>

* Small fix.

Signed-off-by: Tanya <[email protected]>

* Small fix.

Signed-off-by: Tanya <[email protected]>

* Some more refactoring of FWRule minimization code.

Signed-off-by: Tanya <[email protected]>

* Fixed lint error.

Signed-off-by: Tanya <[email protected]>

---------

Signed-off-by: Tanya <[email protected]>

Loading branch information

tanyaveksler authored Jun 9, 2024

1 parent bc23aca commit 0c27129

.github/workflows/test-push.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -87,27 +87,6 @@ jobs: @@
               name: k8s-failed-run-time-check-file
               path: ./tests/k8s_tests_failed_runtime_check.csv
               if-no-files-found: ignore
-      k8s-tests-orig-vs-opt-comparison:
-        runs-on: ubuntu-latest
-        steps:
-          - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
-          - uses: ./.github/actions/setup-nca-env
-          - name: install helm
-            run: |
-              curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
-              chmod 700 get_helm.sh
-              sudo ./get_helm.sh
-          - name: Run k8s tests
-            env:
-              GHE_TOKEN: ${{ github.token }}
-              PYTHONPATH: .
-            run: python tests/run_all_tests.py --type=general --category=k8s --hc_opt=debug | tee tests/k8s_cmp_log.txt ; test ${PIPESTATUS[0]} -eq 0
-          - name: upload run_k8s_tests log
-            if: ${{ always() }}
-            uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
-            with:
-              name: k8s-cmp-log
-              path: tests/k8s_cmp_log.txt
       calico-tests:
         runs-on: ubuntu-latest
         steps:
@@ Expand All / @@ -131,22 +110,6 @@ jobs: @@
               name: calico-failed-run-time-check-file
               path: ./tests/calico_tests_failed_runtime_check.csv
               if-no-files-found: ignore
-      calico-tests-orig-vs-opt-comparison:
-        runs-on: ubuntu-latest
-        steps:
-          - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
-          - uses: ./.github/actions/setup-nca-env
-          - name: Run calico tests
-            env:
-              GHE_TOKEN: ${{ github.token }}
-              PYTHONPATH: .
-            run: python tests/run_all_tests.py --type=general --category=calico --hc_opt=debug | tee tests/calico_cmp_log.txt ; test ${PIPESTATUS[0]} -eq 0
-          - name: upload run_calico_tests log
-            if: ${{ always() }}
-            uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
-            with:
-              name: calico-cmp-log
-              path: tests/calico_cmp_log.txt
       istio-tests:
         runs-on: ubuntu-latest
         steps:
@@ Expand All / @@ -170,22 +133,6 @@ jobs: @@
               name: istio-failed-run-time-check-file
               path: ./tests/istio_tests_failed_runtime_check.csv
               if-no-files-found: ignore
-      istio-tests-orig-vs-opt-comparison:
-        runs-on: ubuntu-latest
-        steps:
-          - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11
-          - uses: ./.github/actions/setup-nca-env
-          - name: Run istio tests
-            env:
-              GHE_TOKEN: ${{ github.token }}
-              PYTHONPATH: .
-            run: python tests/run_all_tests.py --type=general --category=istio --hc_opt=debug | tee tests/istio_cmp_log.txt ; test ${PIPESTATUS[0]} -eq 0
-          - name: upload run_istio_tests log
-            if: ${{ always() }}
-            uses: actions/upload-artifact@5d5d22a31266ced268874388b861e4b58bb5c2f3
-            with:
-              name: istio-cmp-log
-              path: tests/istio_cmp_log.txt
       fw-rules-assertion-tests:
         runs-on: ubuntu-latest
         steps:
@@ Expand Down @@

0 comments on commit `0c27129`

Please sign in to comment.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit

There are no files selected for viewing

0 comments on commit `0c27129`

Commit

There are no files selected for viewing

0 comments on commit 0c27129

0 comments on commit `0c27129`