Merge pull request #66 from CactuseSecurity/develop

Develop

.vscode/launch.json

@@ -82,7 +82,7 @@
             "name": "py-normalizeNwData",
             "type": "debugpy",
             "request": "launch",
-            "program": "${workspaceFolder}/scripts/customizing/modelling/convertNwObjDataExample.py",
+            "program": "${workspaceFolder}/scripts/customizing/modelling/convertNwObjDataFromGitPlain.py",
             "console": "integratedTerminal",
             "env": {
                 "PYTHONPATH": "${PYTHONPATH}:${workspaceRoot}"

CODING_GUIDELINES.md

@@ -27,3 +27,72 @@
 ## C# specific
 - avoid null references
 - methods should be preceded with standard comment header (///)
+
+## Conventional Commits
+The Conventional Commits specification is a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of
+
+### Why Use Conventional Commits
+- Automatically generating CHANGELOGs.
+- Automatically determining a semantic version bump (based on the types of commits landed).
+- Communicating the nature of changes to teammates, the public, and other stakeholders.
+- Triggering build and publish processes.
+- Making it easier for people to contribute to your projects, by allowing them to explore a more structured commit history.
+
+The commit message should be structured as follows:
+```
+<type>[optional scope]: <description>
+[optional body]
+[optional footer(s)]
+```
+
+The commit contains the following structural elements, to communicate intent to the consumers:
+
+- fix: a commit of the type fix patches a bug in your codebase (this correlates with PATCH in Semantic Versioning).
+- feat: a commit of the type feat introduces a new feature to the codebase (this correlates with MINOR in Semantic Versioning).
+- BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:, or appends a ! after the type/scope, introduces a breaking API change (correlating with MAJOR in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type.
+- types other than fix: and feat: are allowed, for example @commitlint/config-conventional (based on the Angular convention) recommends build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others.
+- footers other than BREAKING CHANGE: <description> may be provided and follow a convention similar to git trailer format.
+
+### Examples
+Commit message with description and breaking change footer
+```
+feat: allow provided config object to extend other configs
+
+BREAKING CHANGE: `extends` key in config file is now used for extending other config files
+```
+
+Commit message with ! to draw attention to breaking change
+```
+feat!: send an email to the customer when a product is shipped
+```
+
+Commit message with scope and ! to draw attention to breaking change
+```
+feat(api)!: send an email to the customer when a product is shipped
+```
+
+Commit message with ! to draw attention to breaking change
+```
+chore!: drop support for Node 6
+
+BREAKING CHANGE: use JavaScript features not available in Node 6.
+```
+
+Commit message with no body
+```
+docs: correct spelling of CHANGELOG
+```
+
+Commit message with multi-paragraph body and multiple footers
+```
+fix: prevent racing of requests
+
+Introduce a request id and a reference to latest request. Dismiss
+incoming responses other than from latest request.
+
+Remove timeouts which were used to mitigate the racing issue but are
+obsolete now.
+
+Reviewed-by: Z
+Refs: #123
+```

documentation/revision-history-develop.md

@@ -236,4 +236,21 @@ bugfix release:
 - extra parameters in modelling connection
 
 # 8.5.1 - 18.11.2024 DEVELOP
-- fixing PDF generation
+- reporting - fixing PDF generation on various platforms
+- modelling - fixing AR editing: strict prevention of all area mixing
+
+# 8.5.2 - 27.11.2024 DEVELOP
+- some check point importer fixes
+  - 4 new colors
+  - added Internet object
+  - added voip one more object
+
+# 8.5.3 - 27.11.2024 DEVELOP
+- owner import - make ldap selectable (internal/external)
+- small fixes regarding missing config data for two schedulers (daily, app data import)
+
+# 8.5.4 - 04.12.2024 DEVELOP
+- external request: introduce wait cycles
+
+# 8.6.1 - 12.12.2024 DEVELOP
+- external request: introduce locks

documentation/revision-history-main.md

@@ -435,3 +435,43 @@ Network Modelling feature update
 Fixes
 - various small UI fixes
 - importer (CP: handle None objects)
+
+# 8.6 - 11.12.2024 MAIN
+Features
+- Modelling 
+  - Create Application Zones
+  - Add monitoring for external requests for admins 
+  - Add re-initialization for external requests
+  - consolidation modelling external requests
+  - adding optional access requst on behalf of UI user
+  - adding live update of external task/ticket status
+  - app server name handling rework (NONAME --> <prefix>_<IP address>)
+  - owner groups can now also be external LDAP groups
+
+- Reporting
+  - refining connection report (adding Common service, app role, network area details)
+Fixes
+- Importer
+  - adding missing colors in Check Point importer
+  - new VOIP service object and Internet object
+
+- UI
+  - SECURITY: updating System.Text.Encodings.Web v4.5.0 --> v8.0.0
+
+# 8.6.1 17.12.2024 MAIN
+Fixes network modelling
+- lock external requests to avoid multiple external tickets
+- fix missing comments
+- wait cycles for access request after group changes
+- save publish flag at interface creation
+- disregard dummyAppRole for status determination
+- inherit extra configs from interface
+- sanitize extra configs
+- sort tasks for connection Id and show already adapted name of new members
+- small monitoring adaptations
+- some cleanup + removal of compiler warnings
+- fix ldap group creation regression
+- restrict owner_network uniqness constraint to same import source
+- UI interface search pop-up transformed into filterable table
+
+Upgrade Hasura API to v2.45.1

inventory/group_vars/all.yml

@@ -1,5 +1,5 @@
 ### general settings
-product_version: "8.5.2"
+product_version: "8.6.1"
 ansible_user: "{{ lookup('env', 'USER') }}"
 ansible_become_method: sudo
 ansible_python_interpreter: /usr/bin/python3

inventory/group_vars/apiserver.yml

@@ -8,7 +8,7 @@ api_hasura_admin_test_password: "not4production"
 api_user_email: "{{ api_user }}@{{ api_network_listening_ip_address }}"
 api_home: "{{ fworch_home }}/api"
 api_hasura_cli_bin: "{{ fworch_home }}/api/bin/hasura"
-api_hasura_version: "v2.44.0"
+api_hasura_version: "v2.45.1"
 api_project_name: api
 api_no_metadata: false
 api_rollback_is_running: false

roles/api/files/replace_metadata.json

@@ -5186,20 +5186,22 @@
                   "permission": {
                     "check": {},
                     "columns": [
-                      "id",
-                      "ticket_id",
+                      "create_date",
                       "ext_query_variables",
                       "ext_request_content",
                       "ext_request_state",
                       "ext_request_type",
                       "ext_ticket_id",
                       "ext_ticket_system",
+                      "finish_date",
+                      "id",
                       "last_creation_response",
                       "last_processing_response",
+                      "locked",
                       "owner_id",
                       "task_number",
-                      "create_date",
-                      "finish_date"
+                      "ticket_id",
+                      "wait_cycles"
                     ]
                   },
                   "comment": ""
@@ -5209,17 +5211,19 @@
                   "permission": {
                     "check": {},
                     "columns": [
-                      "id",
-                      "ticket_id",
+                      "create_date",
                       "ext_query_variables",
                       "ext_request_content",
                       "ext_request_state",
                       "ext_request_type",
                       "ext_ticket_system",
+                      "finish_date",
+                      "id",
+                      "locked",
                       "owner_id",
                       "task_number",
-                      "create_date",
-                      "finish_date"
+                      "ticket_id",
+                      "wait_cycles"
                     ]
                   },
                   "comment": ""
@@ -5232,6 +5236,7 @@
                     "columns": [
                       "id",
                       "ticket_id",
+                      "locked",
                       "ext_query_variables",
                       "ext_request_content",
                       "ext_request_state",
@@ -5242,6 +5247,7 @@
                       "last_processing_response",
                       "owner_id",
                       "task_number",
+                      "wait_cycles",
                       "create_date",
                       "finish_date"
                     ],
@@ -5255,6 +5261,7 @@
                     "columns": [
                       "id",
                       "ticket_id",
+                      "locked",
                       "ext_query_variables",
                       "ext_request_content",
                       "ext_request_state",
@@ -5265,6 +5272,7 @@
                       "last_processing_response",
                       "owner_id",
                       "task_number",
+                      "wait_cycles",
                       "create_date",
                       "finish_date"
                     ],
@@ -5276,20 +5284,22 @@
                   "role": "modeller",
                   "permission": {
                     "columns": [
-                      "create_date",
+                      "id",
+                      "ticket_id",
+                      "locked",
                       "ext_query_variables",
                       "ext_request_content",
                       "ext_request_state",
                       "ext_request_type",
                       "ext_ticket_id",
                       "ext_ticket_system",
-                      "finish_date",
-                      "id",
                       "last_creation_response",
                       "last_processing_response",
                       "owner_id",
                       "task_number",
-                      "ticket_id"
+                      "wait_cycles",
+                      "create_date",
+                      "finish_date"
                     ],
                     "filter": {}
                   },
@@ -5303,6 +5313,7 @@
                     "columns": [
                       "id",
                       "ticket_id",
+                      "locked",
                       "ext_query_variables",
                       "ext_request_content",
                       "ext_request_state",
@@ -5313,6 +5324,7 @@
                       "last_processing_response",
                       "owner_id",
                       "task_number",
+                      "wait_cycles",
                       "create_date",
                       "finish_date"
                     ],
@@ -5326,7 +5338,8 @@
                   "permission": {
                     "columns": [
                       "ext_request_state",
-                      "finish_date"
+                      "finish_date",
+                      "locked"
                     ],
                     "filter": {},
                     "check": null

roles/common/tasks/install_syslog.yml

@@ -80,6 +80,7 @@
       if $programname startswith '{{ product_name }}' and $msg contains "Audit" then action(type="omfile" file="{{ fworch_log_dir }}/audit.log" template="fworch")
       if $programname startswith '{{ product_name }}' and $msg contains "FWORCHAlert" then action(type="omfile" file="{{ fworch_log_dir }}/alert.log" template="fworch")
       if $programname == '{{ product_name }}-webhook' then action(type="omfile" file="{{ fworch_log_dir }}/webhook.log" template="fworch")
+      if $programname == 'import-fworch-app-data' then action(type="omfile" file="{{ fworch_log_dir }}/import-fworch-app-data.log" template="fworch")
   become: true
 
 - name: edit logrotate

roles/database/files/sql/creation/fworch-create-constraints.sql

@@ -21,7 +21,7 @@ ALTER TABLE object ADD CONSTRAINT object_obj_ip_not_null CHECK (obj_ip IS NOT NU
 ALTER TABLE object ADD CONSTRAINT object_obj_ip_end_not_null CHECK (obj_ip_end IS NOT NULL OR obj_typ_id=2);
 ALTER TABLE owner ADD CONSTRAINT owner_name_unique_in_tenant UNIQUE ("name","tenant_id");
 ALTER TABLE owner_network ADD CONSTRAINT port_in_valid_range CHECK (port > 0 and port <= 65535);
-ALTER TABLE owner_network ADD CONSTRAINT owner_network_ip_unique UNIQUE (owner_id, ip, ip_end);
+ALTER TABLE owner_network ADD CONSTRAINT owner_network_ip_unique UNIQUE (owner_id, ip, ip_end, import_source);
 ALTER TABLE request.reqelement ADD CONSTRAINT port_in_valid_range CHECK (port > 0 and port <= 65535);
 ALTER TABLE request.implelement ADD CONSTRAINT port_in_valid_range CHECK (port > 0 and port <= 65535);
 Alter Table "rule" add Constraint "rule_altkey" UNIQUE ("dev_id","rule_uid","rule_create",xlate_rule);

roles/database/files/sql/creation/fworch-create-tables.sql

@@ -1110,7 +1110,9 @@ create table ext_request
 	last_creation_response varchar,
 	last_processing_response varchar,
 	create_date Timestamp default now(),
-	finish_date Timestamp
+	finish_date Timestamp,
+	wait_cycles int default 0,
+	locked boolean default false
 );
 
 -- workflow -------------------------------------------------------

roles/database/files/sql/creation/fworch-fill-stm.sql

@@ -117,6 +117,7 @@ insert into config (config_key, config_value, config_user) VALUES ('impChangeNot
 insert into config (config_key, config_value, config_user) VALUES ('impChangeNotifyStartAt', '00:00:00', 0);
 insert into config (config_key, config_value, config_user) VALUES ('externalRequestSleepTime', '0', 0);
 insert into config (config_key, config_value, config_user) VALUES ('externalRequestStartAt', '00:00:00', 0);
+insert into config (config_key, config_value, config_user) VALUES ('externalRequestWaitCycles', '0', 0);
 insert into config (config_key, config_value, config_user) VALUES ('modExtraConfigs', '[]', 0);
 insert into config (config_key, config_value, config_user) VALUES ('extTicketSystems', '[{"Url":"","TicketTemplate":"{\"ticket\":{\"subject\":\"@@TICKET_SUBJECT@@\",\"priority\":\"@@PRIORITY@@\",\"requester\":\"@@ONBEHALF@@\",\"domain_name\":\"\",\"workflow\":{\"name\":\"@@WORKFLOW_NAME@@\"},\"steps\":{\"step\":[{\"name\":\"Erfassung des Antrags\",\"tasks\":{\"task\":{\"fields\":{\"field\":[@@TASKS@@]}}}}]}}}","TasksTemplate":"{\"@xsi.type\":\"multi_access_request\",\"name\":\"GewünschterZugang\",\"read_only\":false,\"access_request\":{\"order\":\"AR1\",\"verifier_result\":{\"status\":\"notrun\"},\"use_topology\":true,\"targets\":{\"target\":{\"@type\":\"ANY\"}},\"users\":{\"user\":@@USERS@@},\"sources\":{\"source\":@@SOURCES@@},\"destinations\":{\"destination\":@@DESTINATIONS@@},\"services\":{\"service\":@@SERVICES@@},\"action\":\"@@ACTION@@\",\"labels\":\"\"}},{\"@xsi.type\":\"text_area\",\"name\":\"Grund für den Antrag\",\"read_only\":false,\"text\":\"@@REASON@@\"},{\"@xsi.type\":\"drop_down_list\",\"name\":\"Regel Log aktivieren?\",\"selection\":\"@@LOGGING@@\"},{\"@xsi.type\":\"date\",\"name\":\"Regel befristen bis:\"},{\"@xsi.type\":\"text_field\",\"name\":\"Anwendungs-ID\",\"text\":\"@@APPID@@\"},{\"@xsi.type\":\"checkbox\",\"name\":\"Die benötigte Kommunikationsverbindung ist im Kommunikationsprofil nach IT-Sicherheitsstandard hinterlegt\",\"value\":@@COM_DOCUMENTED@@},{\"@xsi.type\":\"drop_down_list\",\"name\":\"Expertenmodus: Exakt wie beantragt implementieren (Designervorschlag ignorieren)\",\"selection\":\"Nein\"}"}]', 0);
 insert into config (config_key, config_value, config_user) VALUES ('welcomeMessage', '', 0);