Backend - Add manual webhook execution

[pattisdr]

Is your feature request related to a specific problem?

❗ Dependent on #1224

Add API endpoints and support for manual webhooks in the execution layer.

Describe the solution you'd like

Temporarily caching manual data received

• Add a method to the PrivacyRequest model that caches manual data received for a PolicyManualWebhook. The key might contain a combination of the privacy_request_id and the policy_manual_webhook_id, and store a dictionary of fields/values received.
• Add an API endpoint that lets you save data to the cache (calls the previous method) for a particular privacy request id and policy_manual_webhook with the values corresponding to the fields configured on the PolicyManualWebhook.
  • The request must be a in a "requires_input" state. All this endpoint does is add data to the cache, it doesn't queue the privacy request for processing again.
• Add an API endpoint that lets you get data from the cache for a particular privacy request id and policy manual webhook
• Add an endpoint that shows all the manual webhooks.

Execution

• First thing in privacy execution, add a check that looks up the PolicyManualWebhooks in the database and compares them with the data received from the PolicyManualWebhooks in the cache for the given privacy_request_id. If we don't have entries for all PolicyManualWebhooks, pause request execution.
• Add a separate endpoint for resuming privacy request execution after all the data from the manual webhooks has been received:
  • Request must be in "requires_input" mode.
  • We at least must have entries in the cache for all policy_manual_webhooks, there doesn't need to be data, but we need confirmation that all have them have been approved in that the keys exist in the cache
  • Sets request to in_processing, and queues the privacy request again from the "access step".
• Have upload_access_results separately look up if there's any input data from any PolicyManualWebhooks in the cache and adds those directly to the filtered results with each key as the connection_config key. Unlike regular access results, these fields are just passed directly through, we don't filter on data category.

Describe alternatives you've considered, if any

A description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

[pattisdr]

Remember to update the ERD diagram here along with other docs updates.

Also Manual Webhooks are not associated with Policies as they were laid out in the initial design.

See if resume instructions should be modified with this.

[pattisdr]

Note: Instead of a paused state, these privacy requests are going to be put into a requires_input state

[pattisdr]

Assuming that there is no row inputs added, that there is one value that corresponds. This matches the mockups, but I'm not sure if this will match up in reality. In a pinch, data could be supplied comma-separated.