From 7f487c9cbf8e53bce9002956b0ac17021f37cb1b Mon Sep 17 00:00:00 2001 From: somindatommy Date: Fri, 25 Oct 2019 07:44:38 +0530 Subject: [PATCH 01/53] Fixing formatting errors --- en/docs/setup/migrating-data-publishers.md | 28 +++++++++++----------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/en/docs/setup/migrating-data-publishers.md b/en/docs/setup/migrating-data-publishers.md index e2b4c9f3ee..2c0132368a 100644 --- a/en/docs/setup/migrating-data-publishers.md +++ b/en/docs/setup/migrating-data-publishers.md @@ -3,44 +3,44 @@ In prior versions of WSO2 Identity Server, data publishers were implementations of the [AbstractAuthenticationDataPublisher](https://github.com/wso2-extensions/identity-data-publisher-authentication/blob/master/components/org.wso2.carbon.identity.data.publisher.application.authentication/src/main/java/org/wso2/carbon/identity/data/publisher/application/authentication/AbstractAuthenticationDataPublisher.java) -that are invoked iteratively by the +which were invoked iteratively by the [AuthnDataPublisherProxy](https://github.com/wso2-extensions/identity-data-publisher-authentication/blob/master/components/org.wso2.carbon.identity.data.publisher.application.authentication/src/main/java/org/wso2/carbon/identity/data/publisher/application/authentication/AuthnDataPublisherProxy.java) when a session changes, such that the data publishers send events to -their corresponding destinations. From WSO2 Identity Server 5.8.0 +their corresponding destinations. From WSO2 Identity Server 5.9.0 onwards, all data publishers have been migrated to act as event handlers that subscribe to authentication events. One of the main reasons for this is because the current implementation -causes the DASSessionDataPublisherImpl, AuthenticationAuditLogger and -DASLoginDataPublisherImpl classes to unnecessarily implement all the +causes the _DASSessionDataPublisherImpl, AuthenticationAuditLogger_ and +_DASLoginDataPublisherImpl_ classes to unnecessarily implement all the methods of the -[AbstractAuthenticationDataPublisher](https://github.com/wso2-extensions/identity-data-publisher-authentication/blob/master/components/org.wso2.carbon.identity.data.publisher.application.authentication/src/main/java/org/wso2/carbon/identity/data/publisher/application/authentication/AbstractAuthenticationDataPublisher.java) -. This change provides the capability for the data publishers to +[AbstractAuthenticationDataPublisher](https://github.com/wso2-extensions/identity-data-publisher-authentication/blob/master/components/org.wso2.carbon.identity.data.publisher.application.authentication/src/main/java/org/wso2/carbon/identity/data/publisher/application/authentication/AbstractAuthenticationDataPublisher.java). +This change provides the capability for the data publishers to subscribe only to the relevant events of interest and act upon them. The new design approach is as follows: - When a session changes, the identity-framework publishes an event to - the ` AuthnDataPublisherProxy ` . + the `AuthnDataPublisherProxy`. -- ` AuthnDataPublisherProxy ` uses the - ` IdentityEventService ` in - the identity-framework to invoke corresponding handlers to handle +- `AuthnDataPublisherProxy` uses the `IdentityEventService` in + the identity framework to invoke corresponding handlers to handle the event. - These event handlers extend the [AbstractEventHandler](https://github.com/wso2/carbon-identity-framework/blob/master/components/identity-event/org.wso2.carbon.identity.event/src/main/java/org/wso2/carbon/identity/event/handler/AbstractEventHandler.java) - and override its ` handleEvent ` method. + and override its `handleEvent` method. -The diagrams given below illustrate the difference between the design -approach in versions prior to WSO2 IS 5.8.0 and the new design approach. +The following diagrams illustrates the difference between the design +approach in the versions prior to WSO2 IS 5.9.0 and the new design approach. - **Old design approach** ![](/assets/img/setup/old-design-approach.png) - **New design approach** ![](/assets/img/setup/new-design-approach.png) -This section guides you through migrating an existing data publisher to + +The following section will guide you through migrating an existing data publisher to an event handler. ## Migrating data publisher to event handler From 3f3bbc166f186a76e47139998c41c49acbcc0a32 Mon Sep 17 00:00:00 2001 From: somindatommy Date: Tue, 29 Oct 2019 11:01:08 +0530 Subject: [PATCH 02/53] Addressing requested changes --- en/docs/setup/migrating-data-publishers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/en/docs/setup/migrating-data-publishers.md b/en/docs/setup/migrating-data-publishers.md index 2c0132368a..86b5c823a6 100644 --- a/en/docs/setup/migrating-data-publishers.md +++ b/en/docs/setup/migrating-data-publishers.md @@ -6,7 +6,7 @@ implementations of the which were invoked iteratively by the [AuthnDataPublisherProxy](https://github.com/wso2-extensions/identity-data-publisher-authentication/blob/master/components/org.wso2.carbon.identity.data.publisher.application.authentication/src/main/java/org/wso2/carbon/identity/data/publisher/application/authentication/AuthnDataPublisherProxy.java) when a session changes, such that the data publishers send events to -their corresponding destinations. From WSO2 Identity Server 5.9.0 +their corresponding destinations. From WSO2 Identity Server 5.8.0 onwards, all data publishers have been migrated to act as event handlers that subscribe to authentication events. @@ -31,7 +31,7 @@ The new design approach is as follows: [AbstractEventHandler](https://github.com/wso2/carbon-identity-framework/blob/master/components/identity-event/org.wso2.carbon.identity.event/src/main/java/org/wso2/carbon/identity/event/handler/AbstractEventHandler.java) and override its `handleEvent` method. -The following diagrams illustrates the difference between the design +The following diagrams illustrate the difference between the design approach in the versions prior to WSO2 IS 5.9.0 and the new design approach. - **Old design approach** From e533902c6d4341c637e08e4a31aaf97ed8da5e6f Mon Sep 17 00:00:00 2001 From: somindatommy Date: Tue, 29 Oct 2019 13:30:19 +0530 Subject: [PATCH 03/53] Fixing formatting errors --- en/docs/setup/deployment-patterns.md | 242 ++++++++++++++++++--------- 1 file changed, 159 insertions(+), 83 deletions(-) diff --git a/en/docs/setup/deployment-patterns.md b/en/docs/setup/deployment-patterns.md index c813961cb9..bb424050a3 100644 --- a/en/docs/setup/deployment-patterns.md +++ b/en/docs/setup/deployment-patterns.md @@ -1,20 +1,23 @@ # Deployment Patterns The following sections provide high level information on -the recommended deployment pattern. +the recommended deployment patterns. ### Introduction -You can run multiple nodes of WSO2 Identity Server in a cluster mode to achieve two requirements +You can run multiple nodes of WSO2 Identity Server in a cluster mode to achieve two requirements. -1. If one node becomes unavailable or is experiencing high traffic, another node will seamlessly handle the requests. -2. Balancing traffic handling: Multiple nodes can handle the traffic together so that cluster throughput is higher than the throughput of a single node. +1. Handle requests seamlessly: If one node becomes unavailable or is experiencing high traffic, another node will +seamlessly handle the requests. +2. Balancing traffic handling: Multiple nodes can handle the traffic together so that cluster throughput is higher +than the throughput of a single node. For complete information on clustering concepts, see [Clustering Overview](../../administer/clustering-overview). -The following sections guide you through setting up the deployment pattern, which is an HA Clustered Deployment of 2 WSO2 Identity Server nodes. +The following sections guide you through setting up the deployment pattern, which is an HA Clustered Deployment +of 2 WSO2 Identity Server nodes. -### Deployment prerequisites +## Deployment prerequisites As a first step in planning your deployment, ensure that you have the necessary system requirements and a compatible environment. @@ -71,25 +74,36 @@ necessary system requirements and a compatible environment. -### Configuring Databases +## Configuring Databases -In a clustered deployment, all WSO2 Identity Server nodes are pointed to the same databases to ensure the integrity of the data. And also, you can configure multiple logical databases if you require to keep your data logically separated in the environment. This tutorial demonstrates deployment with an identity database (**IDENTITY_DB**) and a user database (**UM_DB**). +In a clustered deployment, all WSO2 Identity Server nodes are pointed to the same databases to ensure the integrity +of the data. Also, you can configure multiple logical databases if you require to keep your data logically separated +in the environment. Following tutorial demonstrates deployment with an identity database (**IDENTITY_DB**) and a user +database (**UM_DB**). !!! note - Alternatively, you can create more databases for each type of data to separate the data logically. Note that this will NOT make a difference in performance and is not mandatory. Separating databases logically may sometimes help to have a different backup and scaling strategy when the deployment is large and complex. - If you do wish to separate the data logically into separate databases, see [Setting Up Separate Databases for Clustering](../../setup/setting-up-separate-databases-for-clustering). + Alternatively, you can create more databases for each type of data to separate the data logically. + This will **NOT** make a difference in performance and is not mandatory. Separating databases logically may + sometimes help to have a different backup and scaling strategy when the deployment is large and complex. + If you do wish to separate the data logically into separate databases, + see [Setting Up Separate Databases for Clustering](../../setup/setting-up-separate-databases-for-clustering). -The following is a high-level component diagram showing how the system would look like when three databases are used. +The following diagram is a high-level component diagram showing how the system would look like when three databases +are used. ![Component diagram](../assets/img/setup/component-diagram.png) !!! note - For db scripts and more information related to databases, see [Working with databases](../../administer/working-with-databases/). + For db scripts and more information related to databases, see + [Setting up the Physical Database](../../administer/setting-up-the-physical-database). -WSO2 Identity Server uses the `master-datasources.xml` file found in the `/repository/conf/datasources` directory, to configure all databases in a single location. We define data sources that can be referred to in other configuration files as necessary. The code block below shows a sample configuration for a MySQL database. +WSO2 Identity Server uses the `master-datasources.xml` file found in the `/repository/conf/datasources` +directory, to configure all databases in a single location. We define data sources that can be referred to in +other configuration files as necessary. The code block below shows a sample configuration for a MySQL database. +For instructions on how to configure the data sources for other databases, please see, +[Changing the Carbon Database](../../administer/changing-the-carbon-database). - -??? example Click here to view the sample code block +??? example "Click here to view the sample code block" ```xml @@ -159,29 +173,27 @@ WSO2 Identity Server uses the `master-datasources.xml` file found in the ` ``` -!!! note - Observe the jndiConfig element defined with each data source. - ```xml - - jdbc/WSO2CarbonDB - - - jdbc/WSO2UserDS - - … - - jdbc/WSO2IdentityDS - - ``` - These data sources can be referred by this `jndiConfig` Name. - -!!! note - For instructions on how to configure the data sources for other databases, - see [Working with Databases](../../administer/working-with-databases). + + !!! note + Observe the `jndiConfig` element defined with each data source. + ```xml + + jdbc/WSO2CarbonDB + + + jdbc/WSO2UserDS + + … + + jdbc/WSO2IdentityDS + + ``` + These data sources can be referred by this `jndiConfig` Name. -#### Linking identity database +### Linking identity database -As we’ve separated out Identity database and user database, we need to configure them in relevant files. Change the data source name to jdbc/WSO2IdentityDS in `/repository/conf/deployment.toml` +As we’ve separated out Identity database and user database, we need to configure them in relevant files. +Change the data source name to `jdbc/WSO2IdentityDS` in `/repository/conf/deployment.toml` file of both nodes, to configure the new identity database for identity management purposes. ``` @@ -189,18 +201,19 @@ As we’ve separated out Identity database and user database, we need to configu data_source="jdbc/WSO2IdentityDS" ``` -#### Linking user database +### Linking user database -Change the data source name to `jdbc/WSO2UserDS` in `/repository/conf/deployment.toml` file of both nodes, to configure the new user database for user management purposes. +Change the data source name to `jdbc/WSO2UserDS` in `/repository/conf/deployment.toml` file of both +nodes, to configure the new user database for user management purposes. -```xml +``` [realm_manager] data_source = "jdbc/WSO2UserDS" ... ``` -### Mounting the shared registry +## Mounting the shared registry WSO2 Identity Server comprises of three different registry repositories. @@ -208,12 +221,16 @@ WSO2 Identity Server comprises of three different registry repositories. 2. **Configuration Repository**: Used to store product-specific configurations. -3. **Governance Repository**: Used to store configuration and data that are shared across the whole platform. This typically includes services, service descriptions, endpoints or data sources. +3. **Governance Repository**: Used to store configuration and data that are shared across the whole platform. +This typically includes services, service descriptions, endpoints or data sources. !!! info - For more information about the registry, see [Working with the Registry](../../administer/working-with-the-registry). + For more information about the registry, + see [Working with the Registry](../../administer/working-with-the-registry). -In this cluster setup, we use the default h2 database as the local registry in each node individually and the governance and configuration registries should be mounted to share across all nodes. To do this add the following to the file `/repository/conf/registry.xml` in both nodes. +In this cluster setup, we use the default h2 database as the local registry in each node individually. Also, +governance and configuration registries should be mounted to share across all nodes. To do this add the following +to the file `/repository/conf/registry.xml` in both nodes. !!! note We have used the `jndiConfig` name of the identity database to reference the data source. @@ -240,10 +257,14 @@ In this cluster setup, we use the default h2 database as the local registry in e ``` -The local registry information is kept within each node in the in-built h2 database. This is reffered as **WSO2CarbonDB** in the above data sources configuration. There is no harm losing the local registry information and there is no backup requirement on this database. For more information on mounting the registry, see [Sharing Databases in a Cluster](../../administer/sharing-databases-in-a-cluster). +The local registry information is kept within each node in the in-built h2 database. This is reffered as +**WSO2CarbonDB** in the above data sources configuration. There is no harm losing the local registry information +and there is no backup requirement on this database. For more information on mounting the registry, +see [Sharing Databases in a Cluster](../../administer/sharing-databases-in-a-cluster). !!! note - The production recommendation is to set the `` property in the `registry.xml` file to false. This is because the automatic versioning of resources can be an extremely expensive operation. + The production recommendation is to set the `` property in the `registry.xml` + file to false. This is because the automatic versioning of resources can be an extremely expensive operation. `false` To make sure the configurations were applied correctly: @@ -259,11 +280,13 @@ To make sure the configurations were applied correctly: -### Clustering-related configurations +## Clustering-related configurations -Following configurations need to be done to both WSO2 Identity Server nodes in order to enable clustering between them. +Following configurations need to be done to both WSO2 Identity Server nodes in order to enable clustering between +them. -1. Enable clustering on node 1 and node 2 by setting the membership scheme that fits your deployment by editing the `/repository/conf/deployment.toml` file. +1. Enable clustering on node 1 and node 2 by setting the membership scheme that fits your deployment by + editing the `/repository/conf/deployment.toml` file. ``` [clustering] membership_scheme = "wka" @@ -275,7 +298,8 @@ Following configurations need to be done to both WSO2 Identity Server nodes in o - AWS membership scheme - Kubernetes membership scheme - The simplest is the well-known address (WKA) based clustering method. It only suites where all the nodes are deployed on machines having static IP addresses. + The simplest is the well-known address (WKA) based clustering method. It only suites where all the nodes are + deployed on machines having static IP addresses. ``` wka ``` @@ -288,35 +312,55 @@ Following configurations need to be done to both WSO2 Identity Server nodes in o local_member_host = "192.168.2.1" local_member_port = "4000" ``` - Under the `members` section, add the `hostName` and `port` for each WKA member. As we have only two nodes in our sample cluster configuration, we will configure both nodes as WKA nodes. + Under the `members` section, add the `hostName` and `port` for each WKA member. As we have only two nodes + in our sample cluster configuration, we will configure both nodes as WKA nodes. - You can also use IP address ranges for the hostName. For example, `192.168.1.2-10`. This should ensure that the cluster eventually recovers after failures. One shortcoming of doing this is that you can define a range only for the last portion of the IP address. You should also keep in mind that the smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members. + You can also use IP address ranges for the hostName. For example, `192.168.1.2-10`. This should ensure + that the cluster eventually recovers after failures. One shortcoming of doing this is that you can define + a range only for the last portion of the IP address. You should also keep in mind that the smaller + the range, the faster the time it takes to discover members since each node has to scan a lesser + number of potential members. 2. Configure caching. !!! note - From WSO2 Identity Server 5.2.0 onwards, distributed caching is disabled and it is not recommended to use this due to many practical issues that are related to configuring and running distributed caching properly. WSO2 Identity Server employs **Hazelcast** as the primary method of implementing cluster messages while using distributed caching in a simple setup. + From WSO2 Identity Server 5.2.0 onwards, distributed caching is disabled and it is not recommended + to use this due to many practical issues that are related to configuring and running distributed + caching properly. WSO2 Identity Server employs **Hazelcast** as the primary method of implementing + cluster messages while using distributed caching in a simple setup. ??? info "About Caching"
    -
  • Why caching
    Caching is an additional layer on top of databases. It enables to keep the recently used data that are fetched from the database in local memory, so that for subsequent data requests instead of fetching from the database the data can be served from the local memory. Caching has certain advantages and disadvantages that you need to evaluate when deciding on your caching strategy.
    • +
  • Why caching
    Caching is an additional layer on top of the databases. It enables to keep + the recently used data that are fetched from the database in local memory, so that for subsequent + data requests instead of fetching from the database the data can be served from the local memory. + Caching has certain advantages and disadvantages that you need to evaluate when deciding on your + caching strategy.
  • Advantages
      -
    • The load on the underlying database or LDAP is reduced as data is served from already fetched data in memory.
      • -
    • Improved performance due to the reduced number of database calls for repetitive data fetching.
      • +
    • The load on the underlying database or LDAP is reduced as data is served from already + fetched data in memory.
      • +
    • Improved performance due to the reduced number of database calls for repetitive + data fetching.
  • Disadvantages
      -
    • Coherency problems may occur when the data change is not immediately reflected on cached data if one node or an external system updates the database.
      • -
    • Data in memory can become stale yet be served, e.g., serving data from memory while its corresponding record in the database is deleted.
      • +
    • Coherency problems may occur when the data change is not immediately reflected on + cached data if one node or an external system updates the database.
      • +
    • Data in memory can become stale yet be served, e.g., serving data from memory while + its corresponding record in the database is deleted.
??? tip "Caching in WSO2 Identity Server" - Historically WSO2 Identity Server used distributed caching to utilize the above-mentioned advantages as well as to minimize the coherence problem. However, in newer deployment patterns where the network is not tightly controlled, distributed caching fail in unexpected ways. Hence, we **no longer recommend using distributed caching**. Instead, it is **recommended to have local caches** (if required) and **cache invalidation messages** (if required) by considering the information given below. + Historically WSO2 Identity Server used distributed caching to utilize the above-mentioned advantages + as well as to minimize the coherence problem. However, in newer deployment patterns where the network + is not tightly controlled, distributed caching fails in unexpected ways. Hence, we **no longer recommend + using distributed caching**. Instead, it is **recommended to have local caches** (if required) and + **cache invalidation messages** (if required) by considering the information given below.