Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Commit

Permalink
Pause Erasure Execution for Manual Confirmation [#522] (#571)
Browse files Browse the repository at this point in the history 
* First draft - add ability to resume an erasure request by manually confirming the number of rows erased.

* Have the access and erasure endpoints share code.

* Update privacy request pause failure test.

* Add tests for erasure caching.

* Update both get_manual_count and get_manual_erasure_count to be inverses of cache_manual_input and cache_manual_erasure_count.

* Fix some formatting.

* Update changelog and add a docs draft.

* Assert that the access portion of execution isn't called if we submit the privacy request from the erasure step. Assert both access and erasure are called when we submit the privacy request from the access step.

* Fix submit mock.

* Respond to CR.
  • Loading branch information
@pattisdr
pattisdr authored Jun 1, 2022
1 parent a19bb5e commit d3f7ed6
Show file tree
Hide file tree
Showing 18 changed files with 661 additions and 119 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ The types of changes are:
* Added `FIDESOPS__DATABASE__ENABLED` and `FIDESOPS__REDIS__ENABLED` configuration variables to allow `fidesops` to run cleanly in a "stateless" mode without any database or redis cache integration [#550](https://github.com/ethyca/fidesops/pull/550)
* Pause Access Request Execution / Resume on Manual Input in [#554](https://github.com/ethyca/fidesops/pull/554)
* A `[package]` section of the `fidesops.toml` configuration file may specify the path to the `fidesops` package itself [#566](https://github.com/ethyca/fidesops/pull/566)
* Pause Erasure Request Execution / Resume on Manual Input in [#571](https://github.com/ethyca/fidesops/pull/571/)

### Changed
* `MaskingStrategyFactory` and associated `MaskingStrategy` implementations now use a decorator-based registration system, to improve extensibility [#560](https://github.com/ethyca/fidesops/pull/560)
Expand Down
24 changes: 21 additions & 3 deletions docs/fidesops/docs/guides/manual_data.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
In this section we'll cover:

- How to describe a manual dataset
- How to send manual data to resume a privacy request
- How to send manual data to resume the access portion of a privacy request
- How to send manual data to resume the erasure portion of a privacy request

## Overview

Expand Down Expand Up @@ -37,12 +38,14 @@ dataset:
data_type: string
```
## Resuming a paused privacy request
## Resuming a paused access privacy request
A privacy request will pause execution when it reaches a manual collection. An administrator
A privacy request will pause execution when it reaches a manual collection in an access request. An administrator
should manually retrieve the data and send it in a POST request. The fields
should match the fields on the paused collection.
Erasure requests with manual collections will also need data manually added as well.
```json title="<code>POST {{host}}/privacy-request/{{privacy_request_id}}/manual_input</code>"
[{
"box_id": 5,
Expand All @@ -54,4 +57,19 @@ If no manual data can be found, simply pass in an empty list to resume the priva

```json
[]
```

## Resuming a paused erasure privacy request

A privacy request will pause execution when it reaches a manual collection in an erasure request. An administrator
should manually mask the records in question and send confirmation of the rows affected in a POST request.

```json title="<code>POST {{host}}/privacy-request/{{privacy_request_id}}/erasure_confirm</code>"
{"row_count": 2}
```

If no manual data was destroyed, pass in a count of 0 to resume the privacy request:

```json
{"row_count": 0}
```
42 changes: 40 additions & 2 deletions docs/fidesops/docs/postman/Fidesops.postman_collection.json
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"info": {
"_postman_id": "336250bf-42b1-49d2-8b9f-d82706f6f4b4",
"_postman_id": "9019796e-9107-41a1-a268-4d2cda117e85",
"name": "Fidesops",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
Expand Down Expand Up @@ -3526,7 +3526,7 @@
"header": [],
"body": {
"mode": "raw",
"raw": "[\n {\"name\": \"Manual connector\",\n \"key\": \"{{manual_connector}}\",\n \"connection_type\": \"manual\",\n \"access\": \"read\"\n}]",
"raw": "[\n {\"name\": \"Manual connector\",\n \"key\": \"{{manual_connector}}\",\n \"connection_type\": \"manual\",\n \"access\": \"write\"\n}]",
"options": {
"raw": {
"language": "json"
Expand Down Expand Up @@ -3697,6 +3697,44 @@
}
},
"response": []
},
{
"name": "Resume Erasure Request with Confirmed Masked Row Count",
"request": {
"auth": {
"type": "bearer",
"bearer": [
{
"key": "token",
"value": "{{client_token}}",
"type": "string"
}
]
},
"method": "POST",
"header": [],
"body": {
"mode": "raw",
"raw": "{\n \"row_count\": 5\n}",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "{{host}}/privacy-request/{{privacy_request_id}}/erasure_confirm",
"host": [
"{{host}}"
],
"path": [
"privacy-request",
"{{privacy_request_id}}",
"erasure_confirm"
]
}
},
"response": []
}
]
}
Expand Down
145 changes: 105 additions & 40 deletions src/fidesops/api/v1/endpoints/privacy_request_endpoints.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
from fidesops.api.v1.urn_registry import (
PRIVACY_REQUEST_APPROVE,
PRIVACY_REQUEST_DENY,
PRIVACY_REQUEST_MANUAL_ERASURE,
PRIVACY_REQUEST_MANUAL_INPUT,
PRIVACY_REQUEST_RESUME,
REQUEST_PREVIEW,
Expand All @@ -47,7 +48,7 @@
from fidesops.models.client import ClientDetail
from fidesops.models.connectionconfig import ConnectionConfig
from fidesops.models.datasetconfig import DatasetConfig
from fidesops.models.policy import ActionType, Policy, PolicyPreWebhook
from fidesops.models.policy import PausedStep, Policy, PolicyPreWebhook
from fidesops.models.privacy_request import (
ExecutionLog,
PrivacyRequest,
Expand All @@ -64,6 +65,7 @@
PrivacyRequestResponse,
PrivacyRequestVerboseResponse,
ReviewPrivacyRequestIds,
RowCountRequest,
)
from fidesops.service.privacy_request.request_runner_service import PrivacyRequestRunner
from fidesops.service.privacy_request.request_service import (
Expand Down Expand Up @@ -577,23 +579,14 @@ def resume_privacy_request(


def validate_manual_input(
manual_rows: List[Row], collection: CollectionAddress, db: Session
manual_rows: List[Row],
collection: CollectionAddress,
dataset_graph: DatasetGraph,
) -> None:
"""Validate manually-added data for a collection.
The specified collection must exist and all fields must be previously defined.
"""
datasets = DatasetConfig.all(db=db)
dataset_graphs = [dataset_config.get_graph() for dataset_config in datasets]
dataset_graph = DatasetGraph(*dataset_graphs)

node: Optional[Node] = dataset_graph.nodes.get(collection)
if not node:
raise HTTPException(
status_code=HTTP_422_UNPROCESSABLE_ENTITY,
detail=f"Cannot save manual rows. No collection in graph with name: '{collection.value}'.",
)

for row in manual_rows:
for field_name in row:
if not dataset_graph.nodes[collection].contains_field(
Expand All @@ -605,25 +598,15 @@ def validate_manual_input(
)


@router.post(
PRIVACY_REQUEST_MANUAL_INPUT,
status_code=HTTP_200_OK,
response_model=PrivacyRequestResponse,
dependencies=[
Security(verify_oauth_client, scopes=[PRIVACY_REQUEST_CALLBACK_RESUME])
],
)
def resume_with_manual_input(
def resume_privacy_request_with_manual_input(
privacy_request_id: str,
*,
db: Session = Depends(deps.get_db),
cache: FidesopsRedis = Depends(deps.get_cache),
manual_rows: List[Optional[Row]],
) -> PrivacyRequestResponse:
"""Resume a privacy request by passing in manual input for the paused collection.
If there's no manual data to submit, pass in an empty list to resume the privacy request.
"""
db: Session,
cache: FidesopsRedis,
expected_paused_step: PausedStep,
manual_rows: List[Row] = [],
manual_count: Optional[int] = None,
) -> PrivacyRequest:
"""Resume privacy request after validating and caching manual data for an access or an erasure request."""
privacy_request: PrivacyRequest = get_privacy_request_or_error(
db, privacy_request_id
)
Expand All @@ -634,23 +617,49 @@ def resume_with_manual_input(
f"status = {privacy_request.status.value}. Privacy request is not paused.",
)

paused_step: Optional[ActionType]
paused_step: Optional[PausedStep]
paused_collection: Optional[CollectionAddress]
paused_step, paused_collection = privacy_request.get_paused_step_and_collection()
if not paused_collection:
if not paused_collection or not paused_step:
raise HTTPException(
status_code=HTTP_400_BAD_REQUEST,
detail=f"Cannot resume privacy request '{privacy_request.id}'; no paused collection.",
detail=f"Cannot resume privacy request '{privacy_request.id}'; no paused collection or no paused step.",
)

validate_manual_input(manual_rows, paused_collection, db)
logger.info(
f"Caching manual input for privacy request '{privacy_request_id}', collection: '{paused_collection}'"
)
privacy_request.cache_manual_input(paused_collection, manual_rows)
if paused_step != expected_paused_step:
raise HTTPException(
status_code=HTTP_400_BAD_REQUEST,
detail=f"Collection '{paused_collection}' is paused at the {paused_step.value} step. Pass in manual data instead to "
f"'{PRIVACY_REQUEST_MANUAL_ERASURE if paused_step == PausedStep.erasure else PRIVACY_REQUEST_MANUAL_INPUT}' to resume.",
)

datasets = DatasetConfig.all(db=db)
dataset_graphs = [dataset_config.get_graph() for dataset_config in datasets]
dataset_graph = DatasetGraph(*dataset_graphs)

node: Optional[Node] = dataset_graph.nodes.get(paused_collection)
if not node:
raise HTTPException(
status_code=HTTP_422_UNPROCESSABLE_ENTITY,
detail=f"Cannot save manual data. No collection in graph with name: '{paused_collection.value}'.",
)

if paused_step == PausedStep.access:
validate_manual_input(manual_rows, paused_collection, dataset_graph)
logger.info(
f"Caching manual input for privacy request '{privacy_request_id}', collection: '{paused_collection}'"
)
privacy_request.cache_manual_input(paused_collection, manual_rows)

elif paused_step == PausedStep.erasure:
logger.info(
f"Caching manually erased row count for privacy request '{privacy_request_id}', collection: '{paused_collection}'"
)
privacy_request.cache_manual_erasure_count(paused_collection, manual_count)

logger.info(
f"Resuming privacy request '{privacy_request_id}', {paused_step.value} step, from collection '{paused_collection.value}'"
f"Resuming privacy request '{privacy_request_id}', {paused_step.value} step, from collection "
f"'{paused_collection.value}'"
)

privacy_request.status = PrivacyRequestStatus.in_processing
Expand All @@ -664,6 +673,62 @@ def resume_with_manual_input(
return privacy_request


@router.post(
PRIVACY_REQUEST_MANUAL_INPUT,
status_code=HTTP_200_OK,
response_model=PrivacyRequestResponse,
dependencies=[
Security(verify_oauth_client, scopes=[PRIVACY_REQUEST_CALLBACK_RESUME])
],
)
def resume_with_manual_input(
privacy_request_id: str,
*,
db: Session = Depends(deps.get_db),
cache: FidesopsRedis = Depends(deps.get_cache),
manual_rows: List[Row],
) -> PrivacyRequestResponse:
"""Resume a privacy request by passing in manual input for the paused collection.
If there's no manual data to submit, pass in an empty list to resume the privacy request.
"""
return resume_privacy_request_with_manual_input(
privacy_request_id=privacy_request_id,
db=db,
cache=cache,
expected_paused_step=PausedStep.access,
manual_rows=manual_rows,
)


@router.post(
PRIVACY_REQUEST_MANUAL_ERASURE,
status_code=HTTP_200_OK,
response_model=PrivacyRequestResponse,
dependencies=[
Security(verify_oauth_client, scopes=[PRIVACY_REQUEST_CALLBACK_RESUME])
],
)
def resume_with_erasure_confirmation(
privacy_request_id: str,
*,
db: Session = Depends(deps.get_db),
cache: FidesopsRedis = Depends(deps.get_cache),
manual_count: RowCountRequest,
) -> PrivacyRequestResponse:
"""Resume the erasure portion of privacy request by passing in the number of rows that were manually masked.
If no rows were masked, pass in a 0 to resume the privacy request.
"""
return resume_privacy_request_with_manual_input(
privacy_request_id=privacy_request_id,
db=db,
cache=cache,
expected_paused_step=PausedStep.erasure,
manual_count=manual_count.row_count,
)


def review_privacy_request(
db: Session, cache: FidesopsRedis, request_ids: List[str], process_request: Callable
) -> BulkReviewResponse:
Expand Down
1 change: 1 addition & 0 deletions src/fidesops/api/v1/urn_registry.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
REQUEST_STATUS_LOGS = "/privacy-request/{privacy_request_id}/log"
PRIVACY_REQUEST_RESUME = "/privacy-request/{privacy_request_id}/resume"
PRIVACY_REQUEST_MANUAL_INPUT = "/privacy-request/{privacy_request_id}/manual_input"
PRIVACY_REQUEST_MANUAL_ERASURE = "/privacy-request/{privacy_request_id}/erasure_confirm"
REQUEST_PREVIEW = "/privacy-request/preview"

# Rule URLs
Expand Down
5 changes: 5 additions & 0 deletions src/fidesops/models/policy.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@
from fidesops.util.data_category import _validate_data_category


class PausedStep(EnumType):
access = "access"
erasure = "erasure"


class ActionType(str, EnumType):
"""The purpose of a particular privacy request"""

Expand Down
Loading

0 comments on commit d3f7ed6

Please sign in to comment.