Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add missing docs references #1266

Merged
merged 6 commits into from
Oct 7, 2022
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,13 @@ The types of changes are:
* `Fixed` for any bug fixes.
* `Security` in case of vulnerabilities.

## [1.9.2](https://github.com/ethyca/fides/compare/1.9.2...main)
## [Unreleased](https://github.com/ethyca/fides/compare/1.9.2...main)

### Docs

* Add unlinked docs and fix any remaining broken links [#1266]https://github.com/ethyca/fides/pull/1266)

## [1.9.2](https://github.com/ethyca/fides/compare/1.9.1...1.9.2)

### Deprecated

Expand Down
14 changes: 0 additions & 14 deletions docs/fides/docs/development/release_checklist.md

This file was deleted.

89 changes: 0 additions & 89 deletions docs/fides/docs/getting_started.md

This file was deleted.

Empty file.
46 changes: 23 additions & 23 deletions docs/fides/docs/guides/fidesops_workflow.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Fidesops Privacy Request Execution

When a [Privacy Request](privacy_requests.md) is submitted, fidesops performs several prerequisite checks, and then visits your collections in two passes: first, to retrieve relevant data for the subject across all your collections, and again to mask the subject's data, if necessary.
When a [Privacy Request](../getting-started/privacy_requests.md) is submitted, fidesops performs several prerequisite checks, and then visits your collections in two passes: first, to retrieve relevant data for the subject across all your collections, and again to mask the subject's data, if necessary.

The following guide outlines the steps fidesops takes to fulfill a privacy request from end to end, including optional configurations and manual data retrieval.

Expand All @@ -11,9 +11,9 @@ Prior to processing a privacy request, fidesops first creates records to store t
| Step | Description |
| --- | --- |
| **Persist** | Fidesops creates a privacy request in long-term storage to capture high-level information (e.g. date created, current status). Fidesops saves the identity of the subject to both short- and long-term storage. |
| **Verify** | If configured, Fidesops sends an [email](./privacy_requests.md#subject-identity-verification) to the user to verify their identity before proceeding. |
| **Verify** | If configured, Fidesops sends an [email](../getting-started/privacy_requests.md#subject-identity-verification) to the user to verify their identity before proceeding. |
| **Notify** | If configured, the user will receive an [email](./email_communications.md) verifying that their request has been received. |
| **Approve** | If configured, Fidesops will require a system administrator to [approve](./configuration_reference.md) the request before proceeding. |
| **Approve** | If configured, Fidesops will require a system administrator to [approve](../installation/configuration.md) the request before proceeding. |

## Privacy request execution

Expand Down Expand Up @@ -47,50 +47,50 @@ If a request to a pre-execution webhook fails, request execution will exit with


### Access request automation
Access request automation is performed regardless of whether there are access or erasure Rules defined, as both Rules require this data. See how to [configure policies, rules, and rule targets](policies.md) for additional information.
Access request automation is performed regardless of whether there are access or erasure Rules defined, as both Rules require this data. See how to [configure policies, rules, and rule targets](policies.md) for additional information.

This step visits all Collections and retrieves all Fields that you've defined in your [Datasets](datasets.md). Fidesops builds a graph in accordance with how you've designated your Collections are related, visits each Collection in turn, and gathers all the results together.
This step visits all Collections and retrieves all Fields that you've defined in your [Datasets](../getting-started/datasets.md). Fidesops builds a graph in accordance with how you've designated your Collections are related, visits each Collection in turn, and gathers all the results together.

#### Graph building
Fidesops builds a Directed Acyclic Graph, or DAG, where each location or node corresponds to a Collection in one of your Datasets. The graph helps determine the order in which nodes will be visited. Fidesops begins with any Collections that can be queried using the supplied identity data, and then points those Collections toward dependent Collections, etc. If fidesops can't determine how to reach a Collection, it will exit early with a status of `error`. To remedy an errored access request, you update how your Collections are related to each other in your Datasets, and resubmit the privacy request.
Fidesops builds a Directed Acyclic Graph, or DAG, where each location or node corresponds to a Collection in one of your Datasets. The graph helps determine the order in which nodes will be visited. Fidesops begins with any Collections that can be queried using the supplied identity data, and then points those Collections toward dependent Collections, etc. If fidesops can't determine how to reach a Collection, it will exit early with a status of `error`. To remedy an errored access request, you update how your Collections are related to each other in your Datasets, and resubmit the privacy request.

![Access Graph](../img/access_graph.png)

#### Graph Execution
After the graph is built, Fidesops passes the result to [Dask](https://www.dask.org/) to execute sequentially. Fidesops visits one Collection at a time, following the graph created, and uses Dask to determine ordering for ties.
After the graph is built, Fidesops passes the result to [Dask](https://www.dask.org/) to execute sequentially. Fidesops visits one Collection at a time, following the graph created, and uses Dask to determine ordering for ties.

For the first Collections in the graph connected to the root, Fidesops uses the customers' provided identity to locate subject data, by either making database queries or HTTP requests to a configured API endpoint. The details on how to access your data are determined by the [Connection](connection_types.md) type. Fidesops retrieves all Fields that have been configured on the Collection, and caches the results in temporary storage for usage later. Fidesops then passes the results of that Collection to downstream Collections that similarly make queries, temporarily cache the results, and return their results to their own downstream Collections.
For the first Collections in the graph connected to the root, Fidesops uses the customers' provided identity to locate subject data, by either making database queries or HTTP requests to a configured API endpoint. The details on how to access your data are determined by the [Connection](connection_types.md) type. Fidesops retrieves all Fields that have been configured on the Collection, and caches the results in temporary storage for usage later. Fidesops then passes the results of that Collection to downstream Collections that similarly make queries, temporarily cache the results, and return their results to their own downstream Collections.

A Collection isn't visited until Fidesops has searched for data across all of its upstream Collections. This continues until all Collections have been visited. See [Query Execution](query_execution.md) for more information.

![Access Execution](../img/access_execution.png)

If there is a failure trying to retrieve data on any Collections, the request is retried the number of times [configured](./configuration_reference.md) by `task_retry_count` until the request exits with status `error`. Both the `access` step and errored Collection are cached in temporary storage.
If there is a failure trying to retrieve data on any Collections, the request is retried the number of times [configured](../installation/configuration.md) by `task_retry_count` until the request exits with status `error`. Both the `access` step and errored Collection are cached in temporary storage.
Restarting the privacy request will restart from this step and failed Collection. Collections that have already been visited will not be visited again.

#### Final result retrieval
The final step of an automated access request gathers all the results for each Collection from temporary storage.


### Upload results
If configured, Fidesops uploads the results retrieved from access automation for the data subject.
If configured, Fidesops uploads the results retrieved from access automation for the data subject.

For each configured access Rule, Fidesops filter the graph results to match targeted Data Categories. See [Datasets](datasets.md) for more details.
For each configured access Rule, Fidesops filter the graph results to match targeted Data Categories. See [Datasets](../getting-started/datasets.md) for more details.
Fidesops also supplements the results with any data manually uploaded from [manual webhooks](#respond-to-manual-webhooks). Each data package is uploaded in JSON
or CSV format to a specified storage location like Amazon S3. See [Storage](storage.md) for more information.
or CSV format to a specified storage location like Amazon S3. See [Storage](../getting-started/storage.md) for more information.


### Erasure request automation
If applicable, (erasure [Rules](policies.md#Rule-attributes) are configured on your execution policy), Fidesops builds a simpler version of the access request graph, and visits each Collection in turn, performing masking requests as necessary.
If applicable, (erasure [Rules](policies.md#Rule-attributes) are configured on your execution policy), Fidesops builds a simpler version of the access request graph, and visits each Collection in turn, performing masking requests as necessary.

#### Graph building
#### Graph building
The "graph" for an erasure runs on the data from the access request, which is kept in temporary storage, and can be used to locate data for each Collection individually. Because the data has already been found, each Collection could be visited in any order or run in parallel. The graph is configured so each Collection has its previous access request results passed in as inputs, and each Collection returns a count of records masked when complete.

![Erasure Graph](../img/erasure_graph.png)

#### Graph execution
Fidesops visits each Collection sequentially, using a deterministic order set by Dask. For each row of data retrieved in the access request step, Fidesops attempts to mask the data targeting the fields specified on your execution policy, using the [masking strategies](masking_strategies.md) you've defined. If no rows exist from the access request, or no Fields on that Collection match the targeted Data Categories, no masking occurs. Fidesops caches a count of the records
that had fields masked in temporary storage.
Fidesops visits each Collection sequentially, using a deterministic order set by Dask. For each row of data retrieved in the access request step, Fidesops attempts to mask the data targeting the fields specified on your execution policy, using the [masking strategies](masking_strategies.md) you've defined. If no rows exist from the access request, or no Fields on that Collection match the targeted Data Categories, no masking occurs. Fidesops caches a count of the records
that had fields masked in temporary storage.

The masking request might involve an `update` database query or an `update` or `delete` HTTP request depending on the [Connection Type](connection_types.md). The Email Connector type doesn't mask any data itself, but instead persists how to locate and mask that Collection in temporary storage for use later.

Expand All @@ -100,9 +100,9 @@ If masking fails on a given Collection, Fidesops retries the requests for a conf
### Send erasure request emails
After the access and erasure steps have both executed, Fidesops checks if there are any third parties that need to be additionally emailed to complete erasure requests on your behalf. See [emailing third party services to mask data](email_communications.md#Email-third-party-services-to-mask-data) for more information.

Fidesops retrieves any masking instructions cached by Email Connectors in the erasure request step, and combines them into a single email per Dataset.
Fidesops retrieves any masking instructions cached by Email Connectors in the erasure request step, and combines them into a single email per Dataset.

This step is only performed if you have Email Connectors configured. If the email send fails for any reason, the request will exit with status `error`. Fidesops will cache this step in temporary storage, so retrying the request will resume from this point.
This step is only performed if you have Email Connectors configured. If the email send fails for any reason, the request will exit with status `error`. Fidesops will cache this step in temporary storage, so retrying the request will resume from this point.


### Run policy post-execution webhooks
Expand All @@ -112,18 +112,18 @@ If a request to a post-execution webhook fails, request execution will exit with


### Send email notifications
If configured, Fidesops will send a followup email to the data subject to let them know their request has finished processing. For access Rules, the emails will contain links to where the data subject can retrieve data. For erasure Rules, the emails will simplify notify them that their request is complete.
If configured, Fidesops will send a followup email to the data subject to let them know their request has finished processing. For access Rules, the emails will contain links to where the data subject can retrieve data. For erasure Rules, the emails will simplify notify them that their request is complete.

Request execution will then exit with the status `complete`.

## Additional notes
- Fidesops uses Redis as temporary storage to support executing your request. Data automatically retrieved from each Collection, manually uploaded data, and details about where the Privacy Request may be paused or where it failed may all be temporarily stored. This information will expire in accordance with the `FIDESOPS__REDIS__DEFAULT_TTL_SECONDS` [setting](./configuration_reference.md).
- The current fidesops execution strategy prioritizes being able to erase as many of the original Collections requested as possible. If Fidesops masks
- Fidesops uses Redis as temporary storage to support executing your request. Data automatically retrieved from each Collection, manually uploaded data, and details about where the Privacy Request may be paused or where it failed may all be temporarily stored. This information will expire in accordance with the `FIDESOPS__REDIS__DEFAULT_TTL_SECONDS` [setting](../installation/configuration.md).
- The current fidesops execution strategy prioritizes being able to erase as many of the original Collections requested as possible. If Fidesops masks
some Collections and then registers a failure, the current logic will mask the original remaining Collections using the temporarily saved data retrieved in the original access step instead of re-querying the Collections. Once data is masked in one Collection, it could potentially prevent us from being able to locate data in downstream Collections, and so will use temporarily stored data.
- Data added in the interim, or data related to newly added Collections, can be missed.
- If the automated access step fails part of the way through, a new Collection is added, and then the request is restarted from failure,
- If the automated access step fails part of the way through, a new Collection is added, and then the request is restarted from failure,
Fidesops may miss data from already completed Collections downstream, and any Collections further downstream of that set.
- If the erasure step fails, a new Collection is added, and the request is restarted from failure, Fidesops may miss masking data from the new
- If the erasure step fails, a new Collection is added, and the request is restarted from failure, Fidesops may miss masking data from the new
Collection and data downstream of the new Collection.
- Nodes on the graph correspond to individual Collections within Datasets, not Datasets. The graph built may result in Fidesops
visiting a Collection in one Dataset to be able to find data on a Collection in a separate Dataset, which is used to find data on a Collection in the original Dataset.
Expand Down
Loading