Email Connector: Send Email with Erasure Instructions [#1158]

CHANGELOG.md

@@ -31,6 +31,7 @@ The types of changes are:
 * Foundations for a new email connector type [#1142](https://github.com/ethyca/fidesops/pull/1142)
 * Have the new email connector cache action needed for each collection [#1168](https://github.com/ethyca/fidesops/pull/1168)
 * Added `execution_timeframe` to Policy model and schema [#1244](https://github.com/ethyca/fidesops/pull/1244)
+* Wrap up the email connector - it sends an email with erasure instructions as part of request execution [#1246](https://github.com/ethyca/fidesops/pull/1246)
 
 ### Docs
 

docs/fidesops/docs/guides/email_communications.md

@@ -1,12 +1,12 @@
-# Configure Email Communications
-## What is email used for?
+# Configure Automatic Emails
+## What is a fidesops Email Connection?
 
-Fidesops supports email server configurations for sending processing notices to privacy request subjects. Future updates will support outbound email communications with data processors.
+Fidesops supports configuring third party email servers to handle outbound communications.
 
 Supported modes of use:
 
-- Subject Identity Verification - for more information on identity verification in subject requests, see the [Privacy Requests](privacy_requests.md#subject-identity-verification) guide.
-
+- Subject Identity Verification - sends a verification code to the user's email address prior to processing a subject request. for more information on identity verification, see the [Privacy Requests](privacy_requests.md#subject-identity-verification) guide.
+- Erasure Request Email Fulfillment - sends an email to configured third parties to process erasures for a given data subject.  See [creating email Connectors](#email-third-party-services) for more information.
 
 ## Prerequisites
 
@@ -16,12 +16,12 @@ Fidesops currently supports Mailgun for email integrations. Ensure you register
 
     Follow the [Mailgun documentation](https://documentation.mailgun.com/en/latest/api-intro.html#authentication-1) to create a new Domain Sending Key for fidesops. 
 
-    !!! Note 
-        Mailgun automatically generates a **primary account API key** when you sign up for an account. This key allows you to perform all CRUD operations via Mailgun's API endpoints, and for any of your sending domains. For security purposes, using a new **domain sending key** is recommended over your primary API key.
+!!! Note 
+    Mailgun automatically generates a **primary account API key** when you sign up for an account. This key allows you to perform all CRUD operations via Mailgun's API endpoints, and for any of your sending domains. For security purposes, using a new **domain sending key** is recommended over your primary API key.
 
 ## Configuration
 
-### Create the email configuration
+### Create the email config
 
 ```json title="<code>POST api/v1/email/config"
 {
@@ -47,7 +47,7 @@ Fidesops currently supports Mailgun for email integrations. Ensure you register
 
 ### Add the email configuration secrets 
 
-```json title="<code>POST api/v1/email/config/{{email_config_key}}/secret"
+```json title="<code>POST api/v1/email/config/{email_config_key}/secret"
 {
     "mailgun_api_key": "nc123849ycnpq98fnu"
 }
@@ -58,3 +58,102 @@ Fidesops currently supports Mailgun for email integrations. Ensure you register
 |---|----|
 | `mailgun_api_key` | Your Mailgun Domain Sending Key. |
 
+## Email third-party services
+
+Once your email server is configured, you can create an email connector to send automatic erasure requests to third-party services. Fidesops will gather details about each collection described in the connector, and send a single email to the service after all collections have been visited. 
+
+!!! Note
+    Fidesops does not collect confirmation that the erasure was completed by the third party.
+
+
+### Create the connector
+
+Ensure you have created your [email configuration](#configuration) prior to creating a new email connector.
+
+```json title="<code>PATCH api/v1/connection</code>"
+[
+  { 
+    "name": "Email Connection Config",
+    "key": "third_party_email_connector",
+    "connection_type": "email",
+    "access": "write"
+  }
+]
+```
+
+| Field | Description |
+|----|----|
+| `key` | A unique key used to manage your email connector. This is auto-generated from `name` if left blank. Accepted values are alphanumeric, `_`, and `.`. |
+| `name` | A unique user-friendly name for your email connector. |
+| `connection_type` | Must be `email` to create a new email connector. |
+| `access` | Email connectors must be given `write` access in order to send an email. |
+
+
+### Configure notifications
+
+Once your email connector is created, configure any outbound email addresses:
+
+```json title="<code>PUT api/v1/connection/{email_connection_config_key}/secret</code>" 
+{
+    "test_email": "[email protected]",
+    "to_email": "[email protected]"
+}
+```
+
+| Field | Description |
+|----|----|
+| `{email_connection_config_key}` | The unique key that represents the email connection to use. |
+| `to_email` | The user that will be notified via email to complete an erasure request. *Only one `to_email` is supported at this time.* |
+| `test_email` | *Optional.* An email to which you have access for verifying your setup. If your email configuration is working, you will receive an email with mock data similar to the one sent to third-party services. |
+
+### Configure the dataset
+
+Lastly, configure the collections and fields you would like to request be erased or masked. Fidesops will use these fields to compose an email to the third-party service. 
+
+```json title="<code>PUT api/v1/connection/{email_connection_config_key}/dataset" 
+[
+    {
+      "fides_key": "email_dataset",
+      "name": "Dataset not accessible automatically",
+      "description": "Third party data - will email to request erasure",
+      "collections": [
+        {
+          "name": "daycare_customer",
+          "fields": [
+            {
+              "name": "id",
+              "data_categories": [
+                "system.operations"
+              ],
+              "fidesops_meta": {
+                "primary_key": true
+              }
+            },
+            {
+              "name": "child_health_concerns",
+              "data_categories": [
+                "user.biometric_health"
+              ]
+            },
+            {
+              "name": "user_email",
+              "data_categories": [
+                "user.contact.email"
+              ],
+              "fidesops_meta": {
+                "identity": "email"
+              }
+            }
+          ]
+        }
+      ]
+    }
+]
+```
+
+| Field | Description |
+|----|----|
+| `fides_key` | A unique key used to manage your email dataset. This is auto-generated from `name` if left blank. Accepted values are alphanumeric, `_`, and `.`. |
+| `name` | A unique user-friendly name for your email dataset. |
+| `description` | Any additional information used to describe this email dataset. |
+| `collections` | Any collections and associated fields belonging to the third party service, similar to a configured fidesops [Dataset](datasets.md). If you do not know the exact data structure of a third party's database, you can configure a single collection with the fields you would like masked. **Note:** A primary key must be specified on each collection. |

docs/fidesops/docs/guides/privacy_requests.md

@@ -43,13 +43,13 @@ A full list of attributes available to set on the Privacy Request can be found i
 ## Subject Identity Verification 
 
 To have users verify their identity before their Privacy Request is executed, set the `subject_identity_verification_required` 
-variable in your `fidesops.toml` to `TRUE`. You must also set up an EmailConfig that lets Fidesops send automated emails 
+variable in your `fidesops.toml` to `TRUE`. You must also set up an [EmailConfig](./email_communications.md) that lets fidesops send automated emails 
 to your users.
 
-When a user submits a PrivacyRequest, they will be emailed a six-digit code.  They must supply that verification code to Fidesops
+When a user submits a PrivacyRequest, they will be emailed a six-digit code.  They must supply that verification code to fidesops
 to continue privacy request execution.  Until the Privacy Request identity is verified, it will have a status of: `identity_unverified`.
 
-```json title="<code>POST api/v1/privacy-request/<privacy_request_id>/verify</code>"
+```json title="<code>POST api/v1/privacy-request/{privacy_request_id}/verify</code>"
 {"code": "<verification code here>"}
 ```
 

docs/fidesops/docs/guides/reporting.md

@@ -222,7 +222,7 @@ To retrieve information to resume or retry a privacy request, the following endp
 
 ### Paused access request example
 
-The request below is in a `paused` state because we're waiting on manual input from the user to proceed. If we look at the `stopped_collection_details` key, we can see that the request
+The request below is in a `paused` state because we're waiting on manual input from the user to proceed. If we look at the `action_required_details` key, we can see that the request
 paused execution during the `access` step of the `manual_key:filing_cabinet` collection.  The `action_needed.locators` field shows the user they should
 fetch the record in the filing cabinet with a `customer_id` of `72909`, and pull the `authorized_user`, `customer_id`, `id`, and `payment_card_id` fields
 from that record.  These values should be manually uploaded to the `resume_endpoint`.  See the [Manual Data](https://ethyca.github.io/fidesops/guides/manual_data/#resuming-a-paused-access-privacy-request) 
@@ -237,7 +237,7 @@ guides for more information on resuming a paused access request.
             "created_at": "2022-06-06T20:12:28.809815+00:00",
             "started_processing_at": "2022-06-06T20:12:28.986462+00:00",
             ...,
-            "stopped_collection_details": {
+            "action_required_details": {
                 "step": "access",
                 "collection": "manual_key:filing_cabinet",
                 "action_needed": [
@@ -268,7 +268,7 @@ guides for more information on resuming a paused access request.
 
 ### Paused erasure request example
 
-The request below is in a `paused` state because we're waiting on the user to confirm they've masked the appropriate data before proceeding.  The `stopped_collection_details` shows us that the request
+The request below is in a `paused` state because we're waiting on the user to confirm they've masked the appropriate data before proceeding.  The `action_required_details` shows us that the request
 paused execution during the `erasure` step of the `manual_key:filing_cabinet` collection.  Looking at `action_needed.locators` field, we can
 see that the user should find the record in the filing cabinet with an `id` of 2, and replace its `authorized_user` with `None`. 
 A confirmation of the masked records count should be uploaded to the `resume_endpoint`  See the [Manual Data](https://ethyca.github.io/fidesops/guides/manual_data/#resuming-a-paused-erasure-privacy-request) 
@@ -284,7 +284,7 @@ guides for more information on resuming a paused erasure request.
             "finished_processing_at": null,
             "status": "paused",
             ...,
-            "stopped_collection_details": {
+            "action_required_details": {
                 "step": "erasure",
                 "collection": "manual_key:filing_cabinet",
                 "action_needed": [
@@ -325,7 +325,7 @@ After troubleshooting the issues with your postgres connection, you would resume
             "finished_processing_at": null,
             "status": "error",
             ...,
-            "stopped_collection_details": {
+            "action_required_details": {
               "step": "erasure",
               "collection": "postgres_dataset:payment_card",
               "action_needed": null

docs/fidesops/mkdocs.yml

@@ -19,7 +19,6 @@ nav:
           - User Management: ui/user_management.md
   - How-To Guides:
       - View Available Connection Types: guides/connection_types.md
-      - Configure Email Communications: guides/email_communications.md
       - Annotate Complex Fields: guides/complex_fields.md
       - Configure Data Masking: guides/masking_strategies.md
       - Configure Storage Destinations: guides/storage.md
@@ -28,6 +27,7 @@ nav:
       - Configure OneTrust Integration: guides/onetrust.md
       - Preview Query Execution: guides/query_execution.md
       - Data Rights Protocol: guides/data_rights_protocol.md
+      - Configure Automatic Emails: guides/email_communications.md
   - SaaS Connectors:
       - Connect to SaaS Applications: saas_connectors/saas_connectors.md
       - SaaS Configuration: saas_connectors/saas_config.md

src/fidesops/ops/api/v1/endpoints/privacy_request_endpoints.py

@@ -79,7 +79,7 @@
 from fidesops.ops.schemas.privacy_request import (
     BulkPostPrivacyRequests,
     BulkReviewResponse,
-    CollectionActionRequired,
+    CheckpointActionRequired,
     DenyPrivacyRequests,
     ExecutionLogDetailResponse,
     PrivacyRequestCreate,
@@ -492,33 +492,33 @@ def attach_resume_instructions(privacy_request: PrivacyRequest) -> None:
     about how to resume manually if applicable.
     """
     resume_endpoint: Optional[str] = None
-    stopped_collection_details: Optional[CollectionActionRequired] = None
+    action_required_details: Optional[CheckpointActionRequired] = None
 
     if privacy_request.status == PrivacyRequestStatus.paused:
-        stopped_collection_details = privacy_request.get_paused_collection_details()
+        action_required_details = privacy_request.get_paused_collection_details()
 
-        if stopped_collection_details:
+        if action_required_details:
             # Graph is paused on a specific collection
             resume_endpoint = (
                 PRIVACY_REQUEST_MANUAL_ERASURE
-                if stopped_collection_details.step == CurrentStep.erasure
+                if action_required_details.step == CurrentStep.erasure
                 else PRIVACY_REQUEST_MANUAL_INPUT
             )
         else:
             # Graph is paused on a pre-processing webhook
             resume_endpoint = PRIVACY_REQUEST_RESUME
 
     elif privacy_request.status == PrivacyRequestStatus.error:
-        stopped_collection_details = privacy_request.get_failed_collection_details()
+        action_required_details = privacy_request.get_failed_checkpoint_details()
         resume_endpoint = PRIVACY_REQUEST_RETRY
 
-    if stopped_collection_details:
-        stopped_collection_details.step = stopped_collection_details.step.value  # type: ignore
-        stopped_collection_details.collection = (
-            stopped_collection_details.collection.value  # type: ignore
+    if action_required_details:
+        action_required_details.step = action_required_details.step.value  # type: ignore
+        action_required_details.collection = (
+            action_required_details.collection.value if action_required_details.collection else None  # type: ignore
         )
 
-    privacy_request.stopped_collection_details = stopped_collection_details
+    privacy_request.action_required_details = action_required_details
     # replaces the placeholder in the url with the privacy request id
     privacy_request.resume_endpoint = (
         resume_endpoint.format(privacy_request_id=privacy_request.id)
@@ -784,7 +784,10 @@ async def resume_privacy_request_with_manual_input(
     manual_rows: List[Row] = [],
     manual_count: Optional[int] = None,
 ) -> PrivacyRequest:
-    """Resume privacy request after validating and caching manual data for an access or an erasure request."""
+    """Resume privacy request after validating and caching manual data for an access or an erasure request.
+
+    This assumes the privacy request is being resumed from a specific collection in the graph.
+    """
     privacy_request: PrivacyRequest = get_privacy_request_or_error(
         db, privacy_request_id
     )
@@ -796,7 +799,7 @@ async def resume_privacy_request_with_manual_input(
         )
 
     paused_details: Optional[
-        CollectionActionRequired
+        CheckpointActionRequired
     ] = privacy_request.get_paused_collection_details()
     if not paused_details:
         raise HTTPException(
@@ -805,7 +808,7 @@ async def resume_privacy_request_with_manual_input(
         )
 
     paused_step: CurrentStep = paused_details.step
-    paused_collection: CollectionAddress = paused_details.collection
+    paused_collection: Optional[CollectionAddress] = paused_details.collection
 
     if paused_step != expected_paused_step:
         raise HTTPException(
@@ -818,6 +821,12 @@ async def resume_privacy_request_with_manual_input(
     dataset_graphs = [dataset_config.get_graph() for dataset_config in datasets]
     dataset_graph = DatasetGraph(*dataset_graphs)
 
+    if not paused_collection:
+        raise HTTPException(
+            status_code=HTTP_422_UNPROCESSABLE_ENTITY,
+            detail="Cannot save manual data on paused collection. No paused collection saved'.",
+        )
+
     node: Optional[Node] = dataset_graph.nodes.get(paused_collection)
     if not node:
         raise HTTPException(
@@ -939,16 +948,16 @@ async def restart_privacy_request_from_failure(
         )
 
     failed_details: Optional[
-        CollectionActionRequired
-    ] = privacy_request.get_failed_collection_details()
+        CheckpointActionRequired
+    ] = privacy_request.get_failed_checkpoint_details()
     if not failed_details:
         raise HTTPException(
             status_code=HTTP_400_BAD_REQUEST,
             detail=f"Cannot restart privacy request from failure '{privacy_request.id}'; no failed step or collection.",
         )
 
     failed_step: CurrentStep = failed_details.step
-    failed_collection: CollectionAddress = failed_details.collection
+    failed_collection: Optional[CollectionAddress] = failed_details.collection
 
     logger.info(
         "Restarting failed privacy request '%s' from '%s step, 'collection '%s'",
@@ -964,7 +973,7 @@ async def restart_privacy_request_from_failure(
         from_step=failed_step.value,
     )
 
-    privacy_request.cache_failed_collection_details()  # Reset failed step and collection to None
+    privacy_request.cache_failed_checkpoint_details()  # Reset failed step and collection to None
 
     return privacy_request
 

src/fidesops/ops/common_exceptions.py

@@ -101,6 +101,10 @@ class PrivacyRequestPaused(BaseException):
     """Halt Instruction Received on Privacy Request"""
 
 
+class PrivacyRequestErasureEmailSendRequired(BaseException):
+    """Erasure requests will need to be fulfilled by email send.  Exception is raised to change ExecutionLog details"""
+
+
 class SaaSConfigNotFoundException(FidesopsException):
     """Custom Exception - SaaS Config Not Found"""
 

src/fidesops/ops/email_templates/get_email_template.py

@@ -5,6 +5,7 @@
 
 from fidesops.ops.common_exceptions import EmailTemplateUnhandledActionType
 from fidesops.ops.email_templates.template_names import (
+    EMAIL_ERASURE_REQUEST_FULFILLMENT,
     SUBJECT_IDENTITY_VERIFICATION_TEMPLATE,
 )
 from fidesops.ops.schemas.email.email import EmailActionType
@@ -22,6 +23,8 @@
 def get_email_template(action_type: EmailActionType) -> Template:
     if action_type == EmailActionType.SUBJECT_IDENTITY_VERIFICATION:
         return template_env.get_template(SUBJECT_IDENTITY_VERIFICATION_TEMPLATE)
+    if action_type == EmailActionType.EMAIL_ERASURE_REQUEST_FULFILLMENT:
+        return template_env.get_template(EMAIL_ERASURE_REQUEST_FULFILLMENT)
 
     logger.error("No corresponding template linked to the %s", action_type)
     raise EmailTemplateUnhandledActionType(

src/fidesops/ops/email_templates/template_names.py

@@ -1 +1,2 @@
 SUBJECT_IDENTITY_VERIFICATION_TEMPLATE = "subject_identity_verification.html"
+EMAIL_ERASURE_REQUEST_FULFILLMENT = "erasure_request_email_fulfillment.html"