Cache/Surface Resume/Restart Privacy Request Details [#574]

[pattisdr]

❗ Contains migration; check downrev before merging

Purpose

In the event of a paused or failed privacy request, surface to the user how to resume/restart that request.

• Paused privacy requests that are halted at a specific collection will need manual input to proceed. Pause Access Request and Resume with Manual Input [#521] #554 Pause Erasure Execution for Manual Confirmation [#522] #571
• Paused privacy requests due to pre-processing webhooks will need a request to a specific endpoint we passed to them as the callback url. Run Policy Webhooks as part of Privacy Request Execution [#101] #124
• Failed privacy requests will also need to be retried from failure via a specific endpoint. Restart Graph from Failure [#574] #578

Changes

• When a privacy request is paused or fails at a specific collection, cache additional information to aid in restarting: instead of storing a string containing the paused collection and paused step, store a dictionary containing both of these pieces, plus data needed for resuming the request if applicable.
• Added two keys to the privacy request status endpoint for paused/failed privacy requests: "resume_endpoint" and "stopped_collection_details".
• Save "paused_at" when either a pre-execution webhook or a manual collection pauses the request.

A GET /privacy_request/?request_id=<privacy_request_id> now returns something like:

{
    "items": [
        {
            "id": "pri_ed4a6b7d-deab-489a-9a9f-9c2b19cd0713",
            "created_at": "2022-06-06T20:12:28.809815+00:00",
            "started_processing_at": "2022-06-06T20:12:28.986462+00:00",
            "reviewed_at": null,
            "reviewed_by": null,
            "reviewer": null,
            "finished_processing_at": null,
            "status": "paused",
            ...,
            "stopped_collection_details": {
                "step": "access",
                "collection": "manual_key:filing_cabinet",
                "action_needed": [
                    {
                        "locators": {
                            "customer_id": [
                                1
                            ]
                        },
                        "get": [
                            "authorized_user",
                            "customer_id",
                            "id",
                            "payment_card_id"
                        ],
                        "update": null
                    }
                ]
            },
            "resume_endpoint": "/privacy-request/pri_ed4a6b7d-deab-489a-9a9f-9c2b19cd0713/manual_input"
        }
    ],
    "total": 1,
    "page": 1,
    "size": 50
}

A request that needs erasure input looks like this:

{
    "items": [
        {
            "id": "pri_59ea0129-fc6d-4a12-a5bd-2ee647bf5cec",
            "created_at": "2022-06-06T20:22:05.436361+00:00",
            "started_processing_at": "2022-06-06T20:22:05.473280+00:00",
            "reviewed_at": null,
            "reviewed_by": null,
            "reviewer": null,
            "finished_processing_at": null,
            "status": "paused",
            "external_id": null,
            ...,
            "stopped_collection_details": {
                "step": "erasure",
                "collection": "manual_key:filing_cabinet",
                "action_needed": [
                    {
                        "locators": {
                            "id": 2
                        },
                        "get": null,
                        "update": {
                            "authorized_user": null
                        }
                    }
                ]
            },
            "resume_endpoint": "/privacy-request/pri_59ea0129-fc6d-4a12-a5bd-2ee647bf5cec/erasure_confirm"
        }
    ],
    "total": 1,
    "page": 1,
    "size": 50
}

Checklist

• Update CHANGELOG.md file
  • Merge in main so the most recent CHANGELOG.md file is being appended to
  • Add description within the Unreleased section in an appropriate category. Add a new category from the list at the top of the file if the needed one isn't already there.
  • Add a link to this PR at the end of the description with the PR number as the text. example: #1
• Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.
• If docs updated (select one):
  • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
  • documentation issue created (tag docs-team to complete issue separately)
• Good unit test/integration test coverage
• This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
• The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Ticket

Fixes #570

[pattisdr]

thanks for taking the time to review @sanders41! I believe I've addressed your comments.

[pattisdr]

@ethyca/docs-authors I've added a docs draft for how to view details on how to resume or restart your paused and failed privacy requests to the guides.

[sanders41]

LGTM!