Enable Manual Webhooks in Request Execution [#1228] (#1285)

* Add a method to cache data supplied for a manual webhook on a particular privacy request.

* Add an endpoint to retrieve all enabled access manual webhooks.

* Add an endpoint for uploading manual data corresponding to fields in a manual webhook for a given privacy request with "requires_input" status.

* Add an endpoint to view data manually uploaded for an access manual webhook.

- Add new scopes for the endpoints to upload/view manual data for webhooks.
- Enforce that at least one field is added when defining a manual webhook, and add a fallback if no fields were defined.

* Add an endpoint to resume a privacy request from "requires_input" status once all input has been added.  None of the fields are required, but the a key for each manual webhook still needs to exist in the cache to proceed.

As part of request execution check if data has been uploaded (data can be empty) for all manual webhooks. If True, we can proceed with request execution, otherwise, we put the PrivacyRequest in "requires_input" status and exits.

Also adds the manual data uploaded directly to the packet we upload to the user at the very end.

* Update postman collection.

* Fix request_id query param in existing postman request.

* Include additional details about how to resume a "requires_input" privacy request when getting its status.

* Add docs and update changelog.

* Upload new ERD diagram.

* Don't put a privacy request in requires_input state if this policy only has erasure rules.

* Respond to CR!

* Update manual_webhooks.md

CHANGELOG.md

@@ -44,6 +44,7 @@ The types of changes are:
 * Added `due_date` sorting [#1284](https://github.com/ethyca/fidesops/pull/1284)
 * Added erasure endpoints for Shopify connector [#1289](https://github.com/ethyca/fidesops/pull/1289)
 * Adds ability to send email notification upon privacy request completion [#1282](https://github.com/ethyca/fidesops/pull/1282)
+* Enable new manual webhooks in privacy request execution [#1285](https://github.com/ethyca/fidesops/pull/1285)
 
 * Added human readable label to ConnectionType endpoint [#1297](https://github.com/ethyca/fidesops/pull/1297)
 

docs/fidesops/docs/guides/manual_webhooks.md

@@ -0,0 +1,105 @@
+# Manual Webhooks
+
+Manual webhooks are a simple way for data to be manually uploaded for an access request. Erasure requests are not supported at this time.
+They differ from the more complex [manual connection configs](datasets.md#Configure-a-manual-Dataset) that integrate directly with the graph.
+Manual webhooks gather data *outside* of the graph as a first step, and are more similar to [policy_webhooks](policy_webhooks.md).
+
+
+If you have manual webhooks defined, privacy request execution will exit early and remain in a state of `requires_input`.
+Once data has been manually uploaded for all the manual webhooks, then the privacy request can be resumed. Data uploaded 
+for manual webhooks is passed on directly to the data subject alongside the data package. It is
+not filtered on data category.  Any manual data uploaded is passed on as-is.
+
+
+
+## Configuration
+
+### Create a connection config of type `manual_webhook`
+
+
+```json title="<code>POST api/v1/connection</code>"
+[
+    {"name": "Manual Webhook ConnectionConfig",
+    "key": "manual_webhook_key",
+    "connection_type": "manual_webhook",
+    "access": "read"
+    }
+]
+```
+
+| Field | Description |
+|----|----|
+| `key` | *Optional.* A unique key used to manage your connection config. This is auto-generated from `name` if left blank. Accepted values are alphanumeric, `_`, and `.`. |
+| `name` | A unique user-friendly name for your connection config. This key will also be used to identity the manual webhook|
+| `connection_type` | Should be `manual_webhook` for the resource described here. |
+| `access` | One of `read` or `write` |
+
+
+### Define the fields expected for your `manual_webhook`
+
+Submit a list of fields that will need to be manually uploaded.
+
+
+```json title="<code>PATCH api/v1/connection/{{manual_webhook_key}}/access_manual_webhook</code>"
+{
+    "fields": [
+        {"pii_field": "First Name", "dsr_package_label": "first_name"},
+        {"pii_field": "Last Name", "dsr_package_label": "last_name"},
+        {"pii_field": "Phone Number", "dsr_package_label": null},
+        {"pii_field": "Height", "dsr_package_label": "height"}
+    ]
+}
+```
+
+| Field | Description |
+|----|----|
+| `fields` | *Required.* A list of field mappings with `pii_field` and `dsr_package_label` keys. The `pii_field` is the label fidesops will display when it solicits manual input, and the `dsr_package_label` is the identifier fidesops will use when it uploads the data to the data subject.  If no `dsr_package_label` is supplied, it will be created from the `pii_field`.
+
+
+### Upload manual webhook data for a given privacy request
+
+Privacy request execution will exit early with a status of `requires_input` if we're missing data for `manual_webhooks`.
+A request will need to be made for each manual_webhook to upload the requested data before request execution can proceed.
+
+Note that the fields here are dynamic and should match the fields specified on the manual webhook. All fields are optional.
+If no data exists, an empty dictionary should be uploaded. Fidesops treats this upload as confirmation that the
+system was searched for data related to the data subject.
+
+```json title="<code>PATCH /privacy-request/{{privacy_request_id}}/access_manual_webhook/{{manual_webhook_key}}</code>"
+{
+    "first_name": "Jane",
+    "last_name": "Customer"
+}
+```
+
+### Resume Privacy Request Execution
+
+Once a PrivacyRequest with `requires_input` has had all of its manual data uploaded, prompt the privacy request to resume.
+
+```json title="<code>POST /privacy-request/{{privacy_request_id}}/resume_from_requires_input</code>"
+```
+
+#### Example Upload
+
+In this example, we visited one postgres collection automatically and retrieved Jane's `name`, `email`, and `id`.
+Her `first_name` and `last_name` were manually uploaded as part of the `manual_webhook_key` Manual Webhook
+and directly included here.
+
+```json
+
+{
+  "postgres_example:customer": [
+    {
+      "name": "Jane Customer",
+      "email": "[email protected]",
+      "id": 1
+    }
+  ],
+  "manual_webhook_key": [
+    {
+      "first_name": "Jane",
+      "last_name": "Customer"
+    }
+  ]
+}
+```

docs/fidesops/docs/postman/Fidesops.postman_collection.json

@@ -786,7 +786,7 @@
 						"method": "GET",
 						"header": [],
 						"url": {
-							"raw": "{{host}}/privacy-request/?id={{privacy_request_id}}&verbose=True",
+							"raw": "{{host}}/privacy-request/?request_id={{privacy_request_id}}&verbose=True",
 							"host": [
 								"{{host}}"
 							],
@@ -796,7 +796,7 @@
 							],
 							"query": [
 								{
-									"key": "id",
+									"key": "request_id",
 									"value": "{{privacy_request_id}}"
 								},
 								{
@@ -940,6 +940,35 @@
 						}
 					},
 					"response": []
+				},
+				{
+					"name": "Restart failed node",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "POST",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/retry",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"retry"
+							]
+						}
+					},
+					"response": []
 				}
 			]
 		},
@@ -2678,6 +2707,133 @@
 						}
 					},
 					"response": []
+				},
+				{
+					"name": "Get all Access Manual Webhooks",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "GET",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/connection/{{manual_webhook_key}}/access_manual_webhook",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"connection",
+								"{{manual_webhook_key}}",
+								"access_manual_webhook"
+							]
+						}
+					},
+					"response": []
+				},
+				{
+					"name": "Upload Data for Access Manual Webhook",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "PATCH",
+						"header": [],
+						"body": {
+							"mode": "raw",
+							"raw": "{\n    \"first_name\": \"Jane\",\n    \"last_name\": \"Customer\"\n}",
+							"options": {
+								"raw": {
+									"language": "json"
+								}
+							}
+						},
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/access_manual_webhook/{{manual_webhook_key}}",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"access_manual_webhook",
+								"{{manual_webhook_key}}"
+							]
+						}
+					},
+					"response": []
+				},
+				{
+					"name": "View Manually Uploaded Data",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "GET",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/access_manual_webhook/{{manual_webhook_key}}",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"access_manual_webhook",
+								"{{manual_webhook_key}}"
+							]
+						}
+					},
+					"response": []
+				},
+				{
+					"name": "Resume Privacy Request after Manual Input",
+					"request": {
+						"auth": {
+							"type": "bearer",
+							"bearer": [
+								{
+									"key": "token",
+									"value": "{{client_token}}",
+									"type": "string"
+								}
+							]
+						},
+						"method": "POST",
+						"header": [],
+						"url": {
+							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/resume_from_requires_input",
+							"host": [
+								"{{host}}"
+							],
+							"path": [
+								"privacy-request",
+								"{{privacy_request_id}}",
+								"resume_from_requires_input"
+							]
+						}
+					},
+					"response": []
 				}
 			]
 		},
@@ -4169,40 +4325,6 @@
 				}
 			]
 		},
-		{
-			"name": "Restart from Failure",
-			"item": [
-				{
-					"name": "Restart failed node",
-					"request": {
-						"auth": {
-							"type": "bearer",
-							"bearer": [
-								{
-									"key": "token",
-									"value": "{{client_token}}",
-									"type": "string"
-								}
-							]
-						},
-						"method": "POST",
-						"header": [],
-						"url": {
-							"raw": "{{host}}/privacy-request/{{privacy_request_id}}/retry",
-							"host": [
-								"{{host}}"
-							],
-							"path": [
-								"privacy-request",
-								"{{privacy_request_id}}",
-								"retry"
-							]
-						}
-					},
-					"response": []
-				}
-			]
-		},
 		{
 			"name": "ConnectionType",
 			"item": [

docs/fidesops/mkdocs.yml

@@ -28,6 +28,7 @@ nav:
       - Preview Query Execution: guides/query_execution.md
       - Data Rights Protocol: guides/data_rights_protocol.md
       - Configure Automatic Emails: guides/email_communications.md
+      - Configure Manual Webhooks: guides/manual_webhooks.md
   - SaaS Connectors:
       - Connect to SaaS Applications: saas_connectors/saas_connectors.md
       - SaaS Configuration: saas_connectors/saas_config.md

src/fidesops/ops/api/v1/endpoints/manual_webhook_endpoints.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Optional
+from typing import Optional, Sequence
 
 from fastapi import Depends, Security
 from fastapi.encoders import jsonable_encoder
@@ -21,7 +21,11 @@
     WEBHOOK_DELETE,
     WEBHOOK_READ,
 )
-from fidesops.ops.api.v1.urn_registry import ACCESS_MANUAL_WEBHOOK, V1_URL_PREFIX
+from fidesops.ops.api.v1.urn_registry import (
+    ACCESS_MANUAL_WEBHOOK,
+    ACCESS_MANUAL_WEBHOOKS,
+    V1_URL_PREFIX,
+)
 from fidesops.ops.models.connectionconfig import ConnectionConfig, ConnectionType
 from fidesops.ops.models.manual_webhook import AccessManualWebhook
 from fidesops.ops.schemas.manual_webhook_schemas import (
@@ -178,3 +182,21 @@ def delete_access_manual_webhook(
         "Deleted access manual webhook for connection config '%s'",
         connection_config.key,
     )
+
+
+@router.get(
+    ACCESS_MANUAL_WEBHOOKS,
+    status_code=HTTP_200_OK,
+    dependencies=[Security(verify_oauth_client, scopes=[WEBHOOK_READ])],
+    response_model=Sequence[AccessManualWebhookResponse],
+)
+def get_access_manual_webhooks(
+    db: Session = Depends(deps.get_db),
+) -> Sequence[AccessManualWebhookResponse]:
+    """
+    Get all enabled Access Manual Webhooks
+    """
+    logger.info(
+        "Retrieving all enabled access manual webhooks",
+    )
+    return AccessManualWebhook.get_enabled(db)