Cache/Surface Resume/Restart Privacy Request Details [#574]

CHANGELOG.md

@@ -23,6 +23,7 @@ The types of changes are:
 * Subject Request Details page [#563](https://github.com/ethyca/fidesops/pull/563)
 * Restart Graph from Failure [#578](https://github.com/ethyca/fidesops/pull/578)
 * Redis SSL Support [#611](https://github.com/ethyca/fidesops/pull/611)
+* Cache and Surface Resume/Restart Instructions [#591](https://github.com/ethyca/fidesops/pull/591)
 
 ## [1.5.2](https://github.com/ethyca/fidesops/compare/1.5.1...1.5.2)
 

data/dataset/manual_dataset.yml

@@ -10,11 +10,11 @@ dataset:
             fidesops_meta:
               primary_key: true
           - name: authorized_user
-            data_categories: [ user.derived.identifiable ]
+            data_categories: [ user.provided.identifiable ]
             fidesops_meta:
               data_type: string
           - name: customer_id
-            data_categories: [ user.derived.identifiable.unique_id ]
+            data_categories: [ user.provided.identifiable ]
             fidesops_meta:
               references:
                 - dataset: postgres_example_test_dataset

docs/fidesops/docs/guides/reporting.md

@@ -5,6 +5,7 @@ In this section we'll cover:
 - How to check the high-level status of your privacy requests
 - How to get more detailed execution logs of collections and fields that were potentially affected as part of your privacy request.
 - How to download all privacy requests as a CSV
+- How to view details on resuming/restarting a request
 
 Take me directly to [API docs](/fidesops/api#operations-Privacy_Requests-get_request_status_api_v1_privacy_request_get).
 
@@ -183,4 +184,129 @@ To get all privacy requests in CSV format, use the `download_csv` query param:
 ```csv
 Time received,Subject identity,Policy key,Request status,Reviewer,Time approved/denied
 2022-03-14 16:53:28.869258+00:00,{'email': '[email protected]'},my_primary_policy,complete,fid_16ffde2f-613b-4f79-bbae-41420b0f836b,2022-03-14 16:54:08.804283+00:00
+```
+
+## Details to resume a privacy request
+
+A privacy request may pause when manual input is needed from the user, or it might fail for various reason on a specific collection.  
+Details to resume or retry that privacy request can be accessed via the `GET api/v1/privacy-request?request_id=<privacy_request_id>` endpoint.
+
+### Paused Access Request Example
+
+The request below is in a `paused` state because we're waiting on manual input from the user to proceed. If we look at the `stopped_collection_details` key, we can see that the request
+paused execution during the `access` step of the `manual_key:filing_cabinet` collection.  The `action_needed.locators` field shows the user they should
+fetch the record in the filing cabinet with a `customer_id` of `72909`, and pull the `authorized_user`, `customer_id`, `id`, and `payment_card_id` fields
+from that record.  These values should be manually uploaded to the `resume_endpoint`.  See the [Manual Data](https://ethyca.github.io/fidesops/guides/manual_data/#resuming-a-paused-access-privacy-request) 
+guides for more information on resuming a paused access request.
+
+
+```json
+{
+    "items": [
+        {
+            "id": "pri_ed4a6b7d-deab-489a-9a9f-9c2b19cd0713",
+            "created_at": "2022-06-06T20:12:28.809815+00:00",
+            "started_processing_at": "2022-06-06T20:12:28.986462+00:00",
+            ...,
+            "stopped_collection_details": {
+                "step": "access",
+                "collection": "manual_key:filing_cabinet",
+                "action_needed": [
+                    {
+                        "locators": {
+                            "customer_id": [
+                                72909
+                            ]
+                        },
+                        "get": [
+                            "authorized_user",
+                            "customer_id",
+                            "id",
+                            "payment_card_id"
+                        ],
+                        "update": null
+                    }
+                ]
+            },
+            "resume_endpoint": "/privacy-request/pri_ed4a6b7d-deab-489a-9a9f-9c2b19cd0713/manual_input"
+        }
+    ],
+    "total": 1,
+    "page": 1,
+    "size": 50
+}
+```
+
+### Paused Erasure Request Example
+
+The request below is in a `paused` state because we're waiting on the user to confirm they've masked the appropriate data before proceeding.  The `stopped_collection_details` shows us that the request
+paused execution during the `erasure` step of the `manual_key:filing_cabinet` collection.  Looking at `action_needed.locators` field, we can
+see that the user should find the record in the filing cabinet with an `id` of 2, and replace its `authorized_user` with `None`. 
+A confirmation of the masked records count should be uploaded to the `resume_endpoint`  See the [Manual Data](https://ethyca.github.io/fidesops/guides/manual_data/#resuming-a-paused-erasure-privacy-request) 
+guides for more information on resuming a paused erasure request.
+
+```json
+{
+    "items": [
+        {
+            "id": "pri_59ea0129-fc6d-4a12-a5bd-2ee647bf5cec",
+            "created_at": "2022-06-06T20:22:05.436361+00:00",
+            "started_processing_at": "2022-06-06T20:22:05.473280+00:00",
+            "finished_processing_at": null,
+            "status": "paused",
+            ...,
+            "stopped_collection_details": {
+                "step": "erasure",
+                "collection": "manual_key:filing_cabinet",
+                "action_needed": [
+                    {
+                        "locators": {
+                            "id": 2
+                        },
+                        "get": null,
+                        "update": {
+                            "authorized_user": null
+                        }
+                    }
+                ]
+            },
+            "resume_endpoint": "/privacy-request/pri_59ea0129-fc6d-4a12-a5bd-2ee647bf5cec/erasure_confirm"
+        }
+    ],
+    "total": 1,
+    "page": 1,
+    "size": 50
+}
+
+
+```
+
+### Failed Request Example
+
+The below request is an `error` state because something failed in the `erasure` step of the `postgres_dataset:payment_card` collection.  
+After troubleshooting the issues with your postgres connection, you would resume the request with a POST to the `resume_endpoint`.
+
+```json
+{
+    "items": [
+        {
+            "id": "pri_59ea0129-fc6d-4a12-a5bd-2ee647bf5cec",
+            "created_at": "2022-06-06T20:22:05.436361+00:00",
+            "started_processing_at": "2022-06-06T20:22:05.473280+00:00",
+            "finished_processing_at": null,
+            "status": "error",
+            ...,
+            "stopped_collection_details": {
+              "step": "erasure",
+              "collection": "postgres_dataset:payment_card",
+              "action_needed": null
+            },
+            "resume_endpoint": "/privacy-request/pri_59ea0129-fc6d-4a12-a5bd-2ee647bf5cec/retry"
+        }
+    ],
+    "total": 1,
+    "page": 1,
+    "size": 50
+}
+
 ```

src/fidesops/api/v1/endpoints/privacy_request_endpoints.py

@@ -67,6 +67,7 @@
     PrivacyRequestVerboseResponse,
     ReviewPrivacyRequestIds,
     RowCountRequest,
+    StoppedCollection,
 )
 from fidesops.service.privacy_request.request_runner_service import PrivacyRequestRunner
 from fidesops.service.privacy_request.request_service import (
@@ -281,7 +282,6 @@ def execution_logs_by_dataset_name(
 
 def _filter_privacy_request_queryset(
     query: Query,
-    db: Session = Depends(deps.get_db),
     request_id: Optional[str] = None,
     status: Optional[PrivacyRequestStatus] = None,
     created_lt: Optional[datetime] = None,
@@ -306,7 +306,7 @@ def _filter_privacy_request_queryset(
     for end, start, field_name in [
         [created_lt, created_gt, "created"],
         [completed_lt, completed_gt, "completed"],
-        [errored_lt, errored_gt, "errorer"],
+        [errored_lt, errored_gt, "errored"],
         [started_lt, started_gt, "started"],
     ]:
         if end is None or start is None:
@@ -362,6 +362,47 @@ def _filter_privacy_request_queryset(
     return query.order_by(PrivacyRequest.created_at.desc())
 
 
+def attach_resume_instructions(privacy_request: PrivacyRequest) -> None:
+    """
+    Temporarily update a paused or errored privacy request object with instructions from the Redis cache
+    about how to resume manually if applicable.
+    """
+    resume_endpoint: Optional[str] = None
+    stopped_collection_details: Optional[StoppedCollection] = None
+
+    if privacy_request.status == PrivacyRequestStatus.paused:
+        stopped_collection_details = privacy_request.get_paused_collection_details()
+
+        if stopped_collection_details:
+            # Graph is paused on a specific collection
+            resume_endpoint = (
+                PRIVACY_REQUEST_MANUAL_ERASURE
+                if stopped_collection_details.step == PausedStep.erasure
+                else PRIVACY_REQUEST_MANUAL_INPUT
+            )
+        else:
+            # Graph is paused on a pre-processing webhook
+            resume_endpoint = PRIVACY_REQUEST_RESUME
+
+    elif privacy_request.status == PrivacyRequestStatus.error:
+        stopped_collection_details = privacy_request.get_failed_collection_details()
+        resume_endpoint = PRIVACY_REQUEST_RETRY
+
+    if stopped_collection_details:
+        stopped_collection_details.step = stopped_collection_details.step.value
+        stopped_collection_details.collection = (
+            stopped_collection_details.collection.value
+        )
+
+    privacy_request.stopped_collection_details = stopped_collection_details
+    # replaces the placeholder in the url with the privacy request id
+    privacy_request.resume_endpoint = (
+        resume_endpoint.format(privacy_request_id=privacy_request.id)
+        if resume_endpoint
+        else None
+    )
+
+
 @router.get(
     urls.PRIVACY_REQUESTS,
     dependencies=[Security(verify_oauth_client, scopes=[scopes.PRIVACY_REQUEST_READ])],
@@ -401,7 +442,6 @@ def get_request_status(
     query = db.query(PrivacyRequest)
     query = _filter_privacy_request_queryset(
         query,
-        db,
         request_id,
         status,
         created_lt,
@@ -435,6 +475,10 @@ def get_request_status(
         # it is explicitly requested
         for item in paginated.items:
             item.identity = item.get_cached_identity_data()
+            attach_resume_instructions(item)
+    else:
+        for item in paginated.items:
+            attach_resume_instructions(item)
 
     return paginated
 
@@ -618,15 +662,18 @@ def resume_privacy_request_with_manual_input(
             f"status = {privacy_request.status.value}. Privacy request is not paused.",
         )
 
-    paused_step: Optional[PausedStep]
-    paused_collection: Optional[CollectionAddress]
-    paused_step, paused_collection = privacy_request.get_paused_step_and_collection()
-    if not paused_collection or not paused_step:
+    paused_details: Optional[
+        StoppedCollection
+    ] = privacy_request.get_paused_collection_details()
+    if not paused_details:
         raise HTTPException(
             status_code=HTTP_400_BAD_REQUEST,
-            detail=f"Cannot resume privacy request '{privacy_request.id}'; no paused collection or no paused step.",
+            detail=f"Cannot resume privacy request '{privacy_request.id}'; no paused details.",
         )
 
+    paused_step: PausedStep = paused_details.step
+    paused_collection: CollectionAddress = paused_details.collection
+
     if paused_step != expected_paused_step:
         raise HTTPException(
             status_code=HTTP_400_BAD_REQUEST,
@@ -755,15 +802,18 @@ def restart_privacy_request_from_failure(
             detail=f"Cannot restart privacy request from failure: privacy request '{privacy_request.id}' status = {privacy_request.status.value}.",
         )
 
-    failed_step: Optional[PausedStep]
-    failed_collection: Optional[CollectionAddress]
-    failed_step, failed_collection = privacy_request.get_failed_step_and_collection()
-    if not failed_step or not failed_collection:
+    failed_details: Optional[
+        StoppedCollection
+    ] = privacy_request.get_failed_collection_details()
+    if not failed_details:
         raise HTTPException(
             status_code=HTTP_400_BAD_REQUEST,
             detail=f"Cannot restart privacy request from failure '{privacy_request.id}'; no failed step or collection.",
         )
 
+    failed_step: PausedStep = failed_details.step
+    failed_collection: CollectionAddress = failed_details.collection
+
     logger.info(
         f"Restarting failed privacy request '{privacy_request_id}' from '{failed_step} step, 'collection '{failed_collection}'"
     )
@@ -776,7 +826,7 @@ def restart_privacy_request_from_failure(
         privacy_request=privacy_request,
     ).submit(from_step=failed_step)
 
-    privacy_request.cache_failed_step_and_collection()  # Reset failed step and collection to None
+    privacy_request.cache_failed_collection_details()  # Reset failed step and collection to None
 
     return privacy_request