Merge pull request #584 from catenax-ng/docs/arc42/3-3-diagrams

docs(Arc42): add new diagrams for runtime view and deployment view
eclipse-tractusx · Dec 12, 2023 · f2313c0 · f2313c0
2 parents a10a208 + ee94940
commit f2313c0
Show file tree

Hide file tree

Showing 3 changed files with 271 additions and 22 deletions.
diff --git a/docs/arc42/arc42-bpdm-rel_3_2.md → docs/arc42/arc42-bpdm.md b/docs/arc42/arc42-bpdm-rel_3_2.md → docs/arc42/arc42-bpdm.md
@@ -12,15 +12,19 @@
   - [Technical Context](#technical-context)
 - [Solution Strategy (High Level Picture)](#solution-strategy-high-level-picture)
 - [Building Block View](#building-block-view)
-  - [Target Architecture](#target-architecture)
-  - [Architecture for Release 3.2](#architecture-for-release-32)
-  - [Architecture for Release 3.3](#architecture-for-release-33)
+  - [High-Level Architecture (L/S/A Endpoints)](#high-level-architecture-lsa-endpoints)
+  - [High-Level Architecture (Generic Endpoint)](#high-level-architecture-generic-endpoint)
   - [Keycloak Authentication \& Autorization Flow](#keycloak-authentication--autorization-flow)
 - [Runtime View](#runtime-view)
   - [Upload Business Partner (BPN-L)](#upload-business-partner-bpn-l)
+  - [Upsert Generic Business Partner](#upsert-generic-business-partner)
+  - [Update on Golden Record Change](#update-on-golden-record-change)
 - [Deployment View](#deployment-view)
+  - [Applications Deployment without Kubernetes](#applications-deployment-without-kubernetes)
+  - [Single Application Kubernetes Deployment](#single-application-kubernetes-deployment)
 - [Crosscutting Concepts](#crosscutting-concepts)
   - [Business Partner Data Management Standards](#business-partner-data-management-standards)
+  - [Logging Behavior](#logging-behavior)
 - [Architecture Decisions](#architecture-decisions)
 - [Quality Requirements](#quality-requirements)
 - [Risks and Technical Debts](#risks-and-technical-debts)
@@ -41,7 +45,7 @@ The Golden Record business partner data in combination with the BPN acts as the
 > ⚠️ **HINT**: A Business Partner Data cleaning as well as Golden Record Creation Process is **not** part of this reference implementation!
 
 **Additional Information Material**:
-* Visit BPDM on the official Catena-X Website: [bpdm_catenax_website](https://catena-x.net/en/offers/bpdm)
+* Visit BPDM on the official Catena-X Website: [bpdm_catenax_website](https://catena-x.net/en/offers-standards/bpdm)
 
 ## Goals Overview
 
@@ -168,31 +172,34 @@ The following high level view gives a basic overview about the BPDM Components:
 **BPN Issuer**
 * Every participant in the Catena-X network shall have a unique Business Partner Number (BPN) according to the concept defined by the Catena-X BPN concept. The task of the BPN Generator is to issue such a BPN for a presented Business Partner data object. In that, the BPN Generator serves as the central issuing authority for BPNs within Catena-X. 
 * Technically, it constitutes a service that is available as a singleton within the network.
-* Currently (Release 3.2) the BPN Issuer is part of the BPDM Pool. After implementing the BPDM Orchestrator, the BPN Issuer should become an independent component.
+* Currently, creation of BPNs is part of the BPDM Pool implementation. After implementing the BPDM Orchestrator, it can be considered if it should be an independent component.
 
 **BPDM Orchestrator**
-* The BPDM Orchestrator is **not** part of Release 3.2.
 * Intention of the BPDM Orchestrator is to provide a passive component that offers standardized APIs for the BPDM Gate, BPDM Pool and Data Curation and Enrichment Services to orchestrate the process of Golden Record Creation and handling the different states a business partner record can have during this process.
 
 # Building Block View
 
-## Target Architecture
+## High-Level Architecture (L/S/A Endpoints)
 
-![bpdm_target_architecture](../assets/cx_bpdm_target_architecture.drawio.svg)
+Due to a transmission phase there are two concepts of Business Partner Upload Models. The target is to only have the generic Business Partner on the BPDM Gate.
 
-## Architecture for Release 3.2
+![bpdm_current_architecture_LSA](../assets/cx_bpdm_architecture_v3_2.drawio.svg)
 
-![bpdm_current_architecture](../assets/cx_bpdm_architecture_v3_2.drawio.svg)
+
+## High-Level Architecture (Generic Endpoint)
+
+![bpdm_current_architecture_Generic](../assets/cx_bpdm_architecture_v3_3.drawio.svg)
 
 **Simulator Service**
 * To become more independent in testing the BPDM Application, a Simulator Service was developed.
 * The Simulator Services supports the E2E Test Cases to validate the flow from BPDM Gate to BPDM Pool and back again.
 
-## Architecture for Release 3.3
+**EDC Operator**
+* The diagram above shows two EDCs on Operator side. This is only for visualization purpose. On a technical level there is only one EDC.
 
-The transition architecture outlines the next goal for current development phase to get one step closer to the target architecture.
+**SME**
+* Currently there is no SME Application available
 
-In Progress...
 
 
 ## Keycloak Authentication & Autorization Flow
@@ -201,15 +208,15 @@ In Progress...
 ```mermaid
 
 sequenceDiagram
-    participant EDC of CX Member
+    participant BPDM EDC
     participant OpenIDConnect Server
     participant BPDM Gate
 
     autonumber
 
-    EDC of CX Member-->>OpenIDConnect Server: Send Client Credentials
-    OpenIDConnect Server-->>EDC of CX Member: Respond OAuth2 Token
-    EDC of CX Member -->> BPDM Gate: Send Request with OAuth2 Token in Authorization Header
+    BPDM EDC -->>OpenIDConnect Server: Send Client Credentials
+    OpenIDConnect Server-->> BPDM EDC: Respond OAuth2 Token
+    BPDM EDC -->> BPDM Gate: Send Request with OAuth2 Token in Authorization Header
     BPDM Gate -->> OpenIDConnect Server: Validate Token
     OpenIDConnect Server -->> BPDM Gate: Confirms validity of Token
     BPDM Gate -->> BPDM Gate: Check "resource_access" section of OAuth Token
@@ -282,24 +289,263 @@ sequenceDiagram
     end
 ```
 
+## Upsert Generic Business Partner
+
+```mermaid
+sequenceDiagram
+    autonumber
+
+    SharingMember->>Gate: PUT api/catena/input/business-partners <br> Payload: Business Partner Data A
+    Gate-->>Gate: Persist Business Partner Data Input
+    Gate-->>Gate: Set Sharing State to 'Initial'
+    Gate-->>Gate: Add Changelog Entry 'Create' for Business Partner Input
+    Gate->>Orchestrator: POST api/golden-record-tasks <br> Payload: Business Partner Input Data in mode 'UpdateFromSharingMember'
+    Orchestrator-->>Orchestrator: Create Golden Record Task for Business Partner Data
+    Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Result State: 'Pending'
+    Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'CleanAndSync' <br> StepState: 'Queued'
+    Orchestrator-->>Gate: Created Golden Record Task
+    Gate-->>Gate: Set Sharing State <br> Type: 'PENDING' <br> Task ID: Golden Record Task ID
+    Gate-->>SharingMember: Upserted Business Partner
+
+    loop Polling for Step 'CleanAndSync'
+        CleaningServiceDummy->>Orchestrator: POST api/golden-record-tasks/step-reservations <br> Payload: Step 'CleanAndSync'
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'CleanAndSync' <br> StepState: 'Reserved'
+        Orchestrator-->>CleaningServiceDummy: Golden Record Task
+        CleaningServiceDummy-->>CleaningServiceDummy: Set L/S/A and Generic Business Partner Dummy Cleaning Result
+        CleaningServiceDummy-->>CleaningServiceDummy: Set BPN References to L/S/A result
+        CleaningServiceDummy->>Orchestrator: POST api/golden-record-tasks/step-results <br> Payload: Dummy Result
+        Orchestrator-->>Orchestrator: Set Golden Record Task Business Partner Data to Dummy Result
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'PoolSync' <br> StepState: 'Queued'
+        Orchestrator-->>CleaningServiceDummy: Accept
+    end
+
+    loop Polling for Step 'PoolSync'
+        Pool->>Orchestrator: POST api/golden-record-tasks/step-reservations <br> Payload: Step 'PoolSync'
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'PoolSync' <br> StepState: 'Reserved'
+        Orchestrator-->>Pool: Golden Record Task
+        opt Golden Record Legal Entity Data marked as changed
+            Pool-->>Pool: Upsert Legal Entity from Golden Record Task Legal Entity Data
+            Pool-->>Pool: Add Changelog Entry for BPNL
+        end
+        opt Golden Record Site Data marked as changed
+            Pool-->>Pool: Upsert Site from Golden Record Task Site Data
+            Pool-->>Pool: Add Changelog Entry for BPNS
+        end
+         opt Golden Record Address Data marked as changed
+            Pool-->>Pool: Upsert Address from Golden Record Task Address Data
+             Pool-->>Pool: Add Changelog Entry for BPNA
+        end
+        Pool-->>Pool: Set BPNs in Golden Record Task Generic Business Partner Data
+        Pool->>Orchestrator: POST api/golden-record-tasks/step-results <br> Payload: Updated Result
+        Orchestrator-->>Pool: Accept
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'PoolSync' <br> Step State: 'Success'
+         Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Result State: 'Success'
+    end
+    
+    loop Polling for finished Golden Record Tasks
+        Gate-->>Gate: Query sharing states in Sharing State Type 'PENDING'
+        Gate->>Orchestrator: POST golden-record-tasks/state/search <br> Payload: Golde Record Task ID
+        Orchestrator-->Gate: Golden Record Task State and Result
+        Gate-->>Gate: Persist Business Partner Output
+        Gate-->>Gate: Set Sharing State 'Success'
+        Gate-->>Gate: Add Changelog Entry 'Create' for Business Partner Output
+    end
+
+    SharingMember->>Gate: POST api/catena/output/changelog/search <br> Payload: From After Last Search Time
+    Gate-->>SharingMember: Changelog entry with Business Partner External ID
+    SharingMember->>Gate: POST api/catena/output/business-partners/search <br> Payload: External ID
+    Gate-->>SharingMember: Business Partner Output
+```
+
+## Update on Golden Record Change
+
+```mermaid
+sequenceDiagram
+    autonumber
+
+    Pool-->Pool: Add Changelog Entry for BPNL 
+    
+    loop Polling Pool Changelog
+        Gate->>Pool: POST api/catena/changelog/search <br> Payload: From After Last Search Time
+        Pool-->>Gate: Changelog entry for BPNL
+        Gate-->>Gate: Query Business Partner Output with BPNL
+        Gate->>Orchestrator: POST api/golden-record-tasks <br> Payload: Business Partner Output Data in mode 'UpdateFromPool'
+        Orchestrator-->>Orchestrator: Create Golden Record Task for Business Partner Data
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Result State: 'Pending'
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'Clean' <br> StepState: 'Queued'
+        Orchestrator-->>Gate: Created Golden Record Task
+        Gate-->>Gate: Set Sharing State <br> Type: 'PENDING' <br> Task ID: Golden Record Task ID
+    end
+
+    loop Polling for Step 'Clean'
+        CleaningServiceDummy->>Orchestrator: POST api/golden-record-tasks/step-reservations <br> Payload: Step 'Clean'
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'CleanAndSync' <br> StepState: 'Reserved'
+        Orchestrator-->>CleaningServiceDummy: Golden Record Task
+        CleaningServiceDummy->>Orchestrator: POST api/golden-record-tasks/step-results <br> Payload: Golden Record Task Business Partner Data
+        Orchestrator-->>Orchestrator: Set Golden Record Task Business Partner Data to Dummy Result
+        Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Step: 'Clean' <br> Step State: 'Success'
+         Orchestrator-->>Orchestrator: Set Golden Record Task State <br> Result State: 'Success'
+        Orchestrator-->>CleaningServiceDummy: Accept
+    end
+    
+    loop Polling for finished Golden Record Tasks
+        Gate-->>Gate: Query sharing states in Sharing State Type 'PENDING'
+        Gate->>Orchestrator: POST golden-record-tasks/state/search <br> Payload: Golden Record Task ID
+        Orchestrator-->Gate: Golden Record Task State and Result
+        Gate-->>Gate: Persist Business Partner Output
+        Gate-->>Gate: Set Sharing State 'Success'
+        Gate-->>Gate: Add Changelog Entry 'Create' for Business Partner Output
+    end
+    
+    SharingMember->>Gate: POST api/catena/output/changelog/search <br> Payload: From After Last Search Time
+    Gate-->>SharingMember: Changelog entry with Business Partner External ID
+    SharingMember->>Gate: POST api/catena/output/business-partners/search <br> Payload: External ID
+    Gate-->>SharingMember: Business Partner Output
+    
+```
 
 # Deployment View
 
-How to run the service
+## Applications Deployment without Kubernetes
+
+```mermaid
+C4Context
+
+    Person(bpdm_user, "(Technical) User of the BPDM APIs")
+
+    System(pool_postgres, "Pool Database" "Postgres: 14.5")
+    System(gate_postgres, "Gate Database" "Postgres: 14.5")
+    System(bridge_postgres, "Gate Database" "Postgres: 14.5")
 
-![Deployment View](images/deployment-view-3-2.png)
+
+    Deployment_Node(pool_machine, "OS Environment", "Linux Alpine 3.16"){
+        Deployment_Node(pool_java, "Runtime Environment", "JAVA RE 17") {
+            Container(pool_container, "Pool Application", "Spring Boot: 3.1")
+        }
+    }
+
+    Deployment_Node(gate_machine, "OS Environment", "Linux Alpine 3.16"){
+        Deployment_Node(gate_java, "Runtime Environment", "JAVA RE 17") {
+            Container(gate_container, "Gate Application", "Spring Boot: 3.1")
+        }
+    }
+
+    Deployment_Node(bridge_machine, "OS Environment", "Linux Alpine 3.16"){
+        Deployment_Node(bridge_java, "Runtime Environment", "JAVA RE 17") {
+            Container(bridge_container, "Bridge Dummy Application", "Spring Boot: 3.1")
+        }
+    }
+
+    Deployment_Node(orchestrator_machine, "OS Environment", "Linux Alpine 3.16"){
+        Deployment_Node(orchestrator_java, "Runtime Environment", "JAVA RE 17") {
+            Container(orchestrator_container, "Orchestrator Application", "Spring Boot: 3.1")
+        }
+    }
+
+     Deployment_Node(dummy_machine, "OS Environment", "Linux Alpine 3.16"){
+        Deployment_Node(dummy_java, "Runtime Environment", "JAVA RE 17") {
+            Container(dummy_container, "Cleaning Service Dummy Application", "Spring Boot: 3.1")
+        }
+    }
+
+    Rel(bpdm_user, pool_container, "HTTP/S")
+    Rel(pool_container, pool_postgres, "TCP/IP")
+
+    Rel(bpdm_user, gate_container, "HTTP/S")
+    Rel(gate_container, gate_postgres, "TCP/IP")
+
+    Rel(bpdm_user, bridge_container, "HTTP/S")
+    Rel(bridge_container, bridge_postgres, "TCP/IP")
+
+    Rel(pool_container, orchestrator_container, "HTTP/S")
+    Rel(gate_container, orchestrator_container, "HTTP/S")
+    Rel(dummy_container, orchestrator_container, "HTTP/S")
+    
+    Rel(bridge_container, pool_container, "HTTP/S")
+    Rel(bridge_container, gate_container, "HTTP/S")
+
+```
+
+## Single Application Kubernetes Deployment
+
+```mermaid
+C4Context
+
+    Person(bpdm_user, "(Technical) User of the BPDM APIs")
+
+    Deployment_Node(kubernetes, "Kubernetes Environment", "Kubernetes 1.28"){
+
+        Container(ingress, "Ingress", "Ingress Kubernetes Resource")
+        Container(nginx, "Ingress Controller", "Nginx Reverse Proxy")
+        Container(service, "Service", "Service Kubernetes Resource")
+
+        Container(database, "Database Deployment", "Chart bitnami/postgres:11.9.13")
+        Container(other_bpdm, "Other BPDM Application Deployment", "Helm Chart")
+
+        Deployment_Node(deployment, "Deployment", "Deployment Kubernetes Resource"){
+                Deployment_Node(replicaSet_1, "Replica Set", "Ingress ReplicaSet Resource"){
+                    Deployment_Node(pod_1, "Pod", "Pod Kubernetes Resource"){
+                        Container(container_1, "BPDM Application Container", "Spring Boot 3 on Linux Alpine 3.6")
+                        Container(volume_1, "Config Volume", "Kubernetes Volume Mount")
+                    }
+        }
+    }
+
+    Deployment_Node(kubernetes_config, "Kubernetes Configurations", "Logical Grouping"){
+        Container(configMap, "Application Configuration", "Kubernetes ConfigMap Resource")
+        Container(secret, "Secret Configuration", "Kubernetes Secret Resource")
+
+    }
+}
+
+Rel(bpdm_user, nginx, "Sends URL", "HTTPS")
+Rel(ingress, nginx, "Routing Information")
+Rel(nginx, service, "Routes to")
+Rel(service, container_1, "HTTP")
+
+Rel(container_1, volume_1, "mounts")
+Rel(volume_1, configMap, "mounts")
+Rel(volume_1, secret, "mounts")
+
+Rel(container_1, database, "TCP/IP")
+Rel(container_1, other_bpdm, "")
+
+UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="1")
+
+
+
+```
 
 # Crosscutting Concepts
 
 ## Business Partner Data Management Standards
 [bpdm_standards](https://catena-x.net/de/standard-library)
 
+## Logging Behavior
+
+As Spring Boot applications BPDM employs Spring
+specific [logging behavior](https://docs.spring.io/spring-boot/docs/3.0.0/reference/htmlsingle/#features.logging)
+
+We enhance the default log entries with user request information including the determined user ID and a generated request ID.
+Not all logs belong to an ongoing user request in which case these entries are empty.
+
+In addition to the Spring standard logs the BPDM applications keep a log of the following events:
+
+* INFO: User requesting resource with resource name and HTTP verb
+* INFO: Request HTTP response
+* INFO: Update/Create Golden Record Business Partners
+* INFO: Creating BPNs
+* ERROR: Uncaught exceptions occurring in the service logic
+
 # Architecture Decisions
 [Architecture Decision Logs](https://confluence.catena-x.net/display/CORE/BPDM+%7C+Decision+Logs)
 (Link will be replaced with ADRs based on Markdown)
 
 * [001-multitenancy_approach](../decision-records/001-multitenancy_approach.md)
 * [002-edc_for_pool_api](../decision-records/002-edc_for_pool_api.md)
+* [003-orchestrator_serviceApi_vs_messagebus_approach](../decision-records/003-orchestrator_serviceApi_vs_messagebus_approach.md)
+* [004-openapi_descriptions](../decision-records/004-openapi_descriptions.md)
+* [005-edc-usage-for-third-party-services](../decision-records/005-edc-usage-for-third-party-services.md)
 
 # Quality Requirements
 
@@ -314,8 +560,6 @@ How to run the service
 * Will there by a Proxy EDC concept?
 * ...
 
-**Lack on Developer Resources**
-* Too less developer resources in contrast to the expectations that the BPDM Product and its Golden Record will be a foundation component within Catena-X.
 
 **Semantic Model and SSI Integration of the Golden Record**
 * Not in scope.
@@ -327,6 +571,7 @@ How to run the service
 **Data Storage and anonymize concept**
 * How to anonymize the relations between CX-Member and its belonging Business Partner?
 * 💡 Idea: using kind of "ticket numbering"
+* ✔️ Solved via ticketing.
 
 **Accessability for SMEs**
 * Uploading via CSV File. Does it requires an EDC?

diff --git a/docs/assets/cx_bpdm_architecture_v3_2.drawio.svg b/docs/assets/cx_bpdm_architecture_v3_2.drawio.svg
diff --git a/docs/assets/cx_bpdm_architecture_v3_3.drawio.svg b/docs/assets/cx_bpdm_architecture_v3_3.drawio.svg