706 Adds SaaS connection type to SaaS yaml config (#748)

* Adds Saas type to saas yaml config

* To allow data migration that queries on a connectiontype enum to work, update previous schema migrations that used "alter type" to update connection type to rename the enum type, create a new enum with the new types, and then delete the old enum type.

Co-authored-by: Dawn Pattison <[email protected]>

CHANGELOG.md

@@ -22,12 +22,14 @@ The types of changes are:
 * [Admin UI] Update Subject Request status filter to be a multiselect dropdown [#513](https://github.com/ethyca/fidesops/pull/764)
 * Add support for multiple statuses to be selected for filtering subject requests [#660](https://github.com/ethyca/fidesops/pull/802)
 * Erasure support for Zendesk [#775](https://github.com/ethyca/fidesops/pull/775)
+* Adds SaaS connection type to SaaS yaml config [748](https://github.com/ethyca/fidesops/pull/748)
 
 ### Fixed
 * Resolve issue with MyPy seeing files in fidesops as missing imports [#719](https://github.com/ethyca/fidesops/pull/719)
 * Fixed `check-migrations` Make command [#806](https://github.com/ethyca/fidesops/pull/806)
 * Fix issue requiring separate install of snowflake-connector-python [#807](https://github.com/ethyca/fidesops/pull/807)
 
+
 ## [1.6.1](https://github.com/ethyca/fidesops/compare/1.6.0...1.6.1)
 
 ### Added
@@ -52,6 +54,7 @@ The types of changes are:
 * Fix no such container error with docker-compose [#758](https://github.com/ethyca/fidesops/pull/758)
 * Fixed issue with extending the configuration [#721](https://github.com/ethyca/fidesops/pull/721)
 
+
 ## [1.6.0](https://github.com/ethyca/fidesops/compare/1.5.3...1.6.0)
 
 ### Added

data/saas/config/hubspot_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: hubspot_connector_example
   name: Hubspot SaaS Config
+  type: hubspot
   description: A sample schema representing the Hubspot connector for Fidesops
   version: 0.0.1
 

data/saas/config/mailchimp_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: mailchimp_connector_example
   name: Mailchimp SaaS Config
+  type: mailchimp
   description: A sample schema representing the Mailchimp connector for Fidesops
   version: 0.0.1
 

data/saas/config/outreach_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: outreach_connector_example
   name: Outreach Example Config
+  type: outreach
   description: A sample schema representing the Outreach connector for Fidesops
   version: 0.0.1
 

data/saas/config/saas_example_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: saas_connector_example
   name: SaaS Example Config
+  type: custom
   description: A sample schema representing a SaaS connector for Fidesops
   version: 0.0.1
 

data/saas/config/segment_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: segment_connector_example
   name: Segment SaaS Config
+  type: segment
   description: A sample schema representing the Segment connector for Fidesops
   version: 0.0.1
 

data/saas/config/sentry_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: sentry_connector
   name: Sentry SaaS Config
+  type: sentry
   description: A sample schema representing the Sentry connector for Fidesops
   version: 0.0.1
 

data/saas/config/stripe_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: stripe_connector_example
   name: Stripe SaaS Config
+  type: stripe
   description: A sample schema representing the Stripe connector for Fidesops
   version: 0.0.1
 

data/saas/config/zendesk_config.yml

@@ -1,6 +1,7 @@
 saas_config:
   fides_key: zendesk_connector_example
   name: Zendesk SaaS Config
+  type: zendesk
   description: A sample schema representing the Zendesk connector for Fidesops
   version: 0.0.1
 

docs/fidesops/docs/development/contributing_details.md

@@ -92,7 +92,7 @@ Some common Alembic commands are listed below. For a comprehensive guide see: ht
 
 The commands will need to be run inside a shell on your Docker containers, which can be opened with `make server-shell`.
 
-In the `/src` directory:
+In the `/src/fidesops` directory:
 
 - Migrate your database to the latest state: `alembic upgrade head`
 - Get revision id of previous migration: `alembic current`

docs/fidesops/docs/saas_connectors/saas_config.md

@@ -20,6 +20,7 @@ The following is an example SaaS config for Mailchimp:
 saas_config:
   fides_key: mailchimp_connector_example
   name: Mailchimp SaaS Config
+  type: mailchimp
   description: A sample schema representing the Mailchimp connector for Fidesops
   version: 0.0.1
 
@@ -113,6 +114,7 @@ A SaaS config schema contains the following metadata fields:
 
 - `fides_key` Used to uniquely identify the connector, this field is used to link a SaaS config to a dataset.
 - `name` A human-readable name for the connector.
+- `type` Type of SaaS connector. Choose from `hubspot`, `mailchimp`, `outreach`, `segment`, `sentry`, `stripe`, `zendesk` or use `custom` for other types.
 - `description` Used to add a useful description.
 - `version` Used to track different versions of the SaaS config.
 

docs/fidesops/docs/saas_connectors/saas_connectors.md

@@ -44,6 +44,7 @@ PATCH api/v1/connection/{saas_key}/saas_config
 {
     "fides_key": "mailchimp_connector_example",
     "name": "Mailchimp SaaS Config",
+    "type": "mailchimp",
     "description": "A sample schema representing the Mailchimp connector for Fidesops"
     ...
 ```

src/fidesops/migrations/versions/07014ff34eb2_add_mariadb.py

@@ -16,7 +16,17 @@
 
 
 def upgrade():
-    op.execute("alter type connectiontype add value 'mariadb'")
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'snowflake', 'redshift', 'mssql', 'mariadb')"
+    )
+    op.execute(
+        (
+            "alter table connectionconfig alter column connection_type type connectiontype using "
+            "connection_type::text::connectiontype"
+        )
+    )
+    op.execute("drop type connectiontype_old")
 
 
 def downgrade():

src/fidesops/migrations/versions/1dfc5a2d30e7_add_saas_config_to_connection_config.py

@@ -23,7 +23,18 @@ def upgrade():
             "saas_config", postgresql.JSONB(astext_type=sa.Text()), nullable=True
         ),
     )
-    op.execute("alter type connectiontype add value 'saas'")
+
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'snowflake', 'redshift', 'mssql', 'mariadb', 'saas')"
+    )
+    op.execute(
+        (
+            "alter table connectionconfig alter column connection_type type connectiontype using "
+            "connection_type::text::connectiontype"
+        )
+    )
+    op.execute("drop type connectiontype_old")
 
 
 def downgrade():

src/fidesops/migrations/versions/3a7c5fb119c9_add_manual_connector_type.py

@@ -16,7 +16,17 @@
 
 
 def upgrade():
-    op.execute("alter type connectiontype add value 'manual'")
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'snowflake', 'redshift', 'mssql', 'mariadb', 'bigquery', 'saas', 'manual')"
+    )
+    op.execute(
+        (
+            "alter table connectionconfig alter column connection_type type connectiontype using "
+            "connection_type::text::connectiontype"
+        )
+    )
+    op.execute("drop type connectiontype_old")
     op.execute("alter type executionlogstatus add value 'paused'")
 
 

src/fidesops/migrations/versions/e55a51b354e3_add_bigquery.py

@@ -16,7 +16,17 @@
 
 
 def upgrade():
-    op.execute("alter type connectiontype add value 'bigquery'")
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'snowflake', 'redshift', 'mssql', 'mariadb', 'bigquery')"
+    )
+    op.execute(
+        (
+            "alter table connectionconfig alter column connection_type type connectiontype using "
+            "connection_type::text::connectiontype"
+        )
+    )
+    op.execute("drop type connectiontype_old")
 
 
 def downgrade():

src/fidesops/migrations/versions/f206d4e7574d_add_redshift_and_snowflake_support.py

@@ -15,8 +15,17 @@
 
 
 def upgrade():
-    op.execute("alter type connectiontype add value 'redshift'")
-    op.execute("alter type connectiontype add value 'snowflake'")
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'redshift', 'snowflake')"
+    )
+    op.execute(
+        (
+            "alter table connectionconfig alter column connection_type type connectiontype using "
+            "connection_type::text::connectiontype"
+        )
+    )
+    op.execute("drop type connectiontype_old")
 
 
 def downgrade():

src/fidesops/migrations/versions/f3841942d90c_add_mssql.py

@@ -16,7 +16,17 @@
 
 
 def upgrade():
-    op.execute("alter type connectiontype add value 'mssql'")
+    op.execute("alter type connectiontype rename to connectiontype_old")
+    op.execute(
+        "create type connectiontype as enum('postgres', 'mongodb', 'mysql', 'https', 'snowflake', 'redshift', 'mssql')"
+    )
+    op.execute(
+        (
+            "alter table connectionconfig alter column connection_type type connectiontype using "
+            "connection_type::text::connectiontype"
+        )
+    )
+    op.execute("drop type connectiontype_old")
 
 
 def downgrade():

src/fidesops/migrations/versions/fc90277bbcde_populate_saas_type_to_saas_config.py

@@ -0,0 +1,76 @@
+"""Populate saas type to saas config
+
+Revision ID: fc90277bbcde
+Revises: ed1b00ff963d
+Create Date: 2022-07-05 19:20:59.384767
+
+"""
+import enum
+from typing import List
+
+from alembic import op
+from sqlalchemy import text
+
+# revision identifiers, used by Alembic.
+revision = "fc90277bbcde"
+down_revision = "c7cc36820d4b"
+branch_labels = None
+depends_on = None
+
+
+class SaaSType(enum.Enum):
+    """
+    Enum to store saas connection type in Fidesops
+    """
+
+    mailchimp = "mailchimp"
+    hubspot = "hubspot"
+    outreach = "outreach"
+    segment = "segment"
+    sentry = "sentry"
+    stripe = "stripe"
+    zendesk = "zendesk"
+    custom = "custom"
+
+
+query = text(
+    """select connectionconfig.id, connectionconfig.saas_config from connectionconfig WHERE connectionconfig.connection_type = :connection_type"""
+)
+update_query = text(
+    """update connectionconfig set saas_config = jsonb_set(saas_config, '{type}', :saas_type) where id = :id"""
+)
+
+
+def upgrade():
+    connection = op.get_bind()
+    saas_options: List[str] = [saas_type.value for saas_type in SaaSType]
+    for id, saas_config in connection.execute(query, {"connection_type": "saas"}):
+        if not saas_config:
+            continue
+        fides_key: str = saas_config.get("fides_key", "")
+        saas_name: str = saas_config.get("name", "")
+
+        try:
+            saas_type: str = next(
+                (
+                    opt
+                    for opt in saas_options
+                    if any(
+                        [
+                            fides_key.lower().startswith(opt),
+                            saas_name.lower().startswith(opt),
+                        ]
+                    )
+                ),
+                "custom",
+            )
+            connection.execute(update_query, {"saas_type": f'"{saas_type}"', "id": id})
+        except Exception:
+            # default to using "custom" if something went wrong
+            connection.execute(update_query, {"saas_type": f'"custom"', "id": id})
+
+
+def downgrade():
+    connection = op.get_bind()
+    for id, saas_config in connection.execute(query, {"connection_type": "saas"}):
+        connection.execute(update_query, {"saas_type": f'"custom"', "id": id})

src/fidesops/schemas/saas/saas_config.py

@@ -1,3 +1,4 @@
+from enum import Enum
 from typing import Any, Dict, List, Literal, Optional, Set, Union
 
 from pydantic import BaseModel, Extra, root_validator, validator
@@ -175,6 +176,21 @@ class ConnectorParam(BaseModel):
     description: Optional[str]
 
 
+class SaaSType(Enum):
+    """
+    Enum to store saas connection type in Fidesops
+    """
+
+    mailchimp = "mailchimp"
+    hubspot = "hubspot"
+    outreach = "outreach"
+    segment = "segment"
+    sentry = "sentry"
+    stripe = "stripe"
+    zendesk = "zendesk"
+    custom = "custom"
+
+
 class SaaSConfig(BaseModel):
     """
     Used to store endpoint and param configurations for a SaaS connector.
@@ -188,6 +204,7 @@ class SaaSConfig(BaseModel):
 
     fides_key: FidesOpsKey
     name: str
+    type: SaaSType
     description: str
     version: str
     connector_params: List[ConnectorParam]
@@ -196,6 +213,11 @@ class SaaSConfig(BaseModel):
     test_request: SaaSRequest
     data_protection_request: Optional[SaaSRequest] = None  # GDPR Delete
 
+    @validator("type", pre=True)
+    def lowercase_saas_type(cls, value: str) -> str:
+        """Enforce lowercase on saas type."""
+        return value.lower()
+
     @property
     def top_level_endpoint_dict(self) -> Dict[str, Endpoint]:
         """Returns a map of endpoint names mapped to Endpoints"""
@@ -241,6 +263,11 @@ def get_graph(self) -> Dataset:
             connection_key=self.fides_key,
         )
 
+    class Config:
+        """Populate models with the raw value of enum fields, rather than the enum itself"""
+
+        use_enum_values = True
+
 
 class SaaSConfigValidationDetails(BaseSchema):
     """

tests/api/v1/endpoints/test_saas_config_endpoints.py

@@ -319,6 +319,7 @@ def test_get_saas_config(
             == saas_example_connection_config.get_saas_config().fides_key
         )
         assert len(response_body["endpoints"]) == 7
+        assert response_body["type"] == "custom"
 
 
 @pytest.mark.unit_saas

tests/fixtures/saas_example_fixtures.py

@@ -200,6 +200,7 @@ def oauth2_connection_config(db: Session, oauth2_configuration) -> Generator:
     saas_config = {
         "fides_key": "oauth2_connector",
         "name": "OAuth2 Connector",
+        "type": "custom",
         "description": "Generic OAuth2 connector for testing",
         "version": "0.0.1",
         "connector_params": [{"name": item} for item in secrets.keys()],