[#1241] Restore default DSR policies

[seanpreston]

Closes #1241

Code Changes

• This PR adds a method load_default_policies to be run within init_db
• load_default_policies creates two DSR execution policies equivalent to those in Fidesops

Steps to Confirm

• nox -s teardown -- volumes
• nox -s dev
• docker compose run fides /bin/bash, then:

pip install ipython
ipython

and in the shell prompt (this can be copied and pasted)

# All table models must be loaded in to assuage any dependency issues
from fideslib.db.base_class import Base
from fideslib.models.audit_log import AuditLog
from fideslib.models.client import ClientDetail
from fideslib.models.fides_user import FidesUser
from fideslib.models.fides_user_permissions import FidesUserPermissions
from fides.api.ops.models.authentication_request import AuthenticationRequest
from fides.api.ops.models.connectionconfig import ConnectionConfig
from fides.api.ops.models.datasetconfig import DatasetConfig
from fides.api.ops.models.email import EmailConfig
from fides.api.ops.models.manual_webhook import AccessManualWebhook
from fides.api.ops.models.policy import Policy, Rule, RuleTarget
from fides.api.ops.models.privacy_request import PrivacyRequest
from fides.api.ops.models.storage import StorageConfig

# Create a database session
from fides.api.ctl.database.session import sync_session
db = sync_session()

# Do some assertions on data that should have been loaded in during `nox -s dev`
assert len(Policy.all(db=db)) == 2
assert len(Rule.all(db=db)) == 2
assert len(RuleTarget.all(db=db)) == 62

if no AssertionErrors or otherwise are thrown, this script has worked

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation Updated:
  • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
  • documentation issue created (tag docs-team to complete issue separately)
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md

[NevilleS]

LGTM, but with several nits.

[ThomasLaPiana]

@seanpreston when I run the server, I'm seeing this error:

2022-10-13 10:05:44.698 [ERROR] (database:configure_db:101): Unable to configure database: fides.api.ops.common_exceptions.PolicyValidationError: Policy rules are invalid, action conflict in erasure rules detected for categories user.name and user

[ThomasLaPiana]

also it appears the database reset aren't working either

Edit: turns it out was related. I commented out the "erasure rules" setup code chunk and it works just fine now, including database resets. This is the offending section:

    log.info("Creating: Ops Data Category Erasure Rules...")
    for target in default_data_categories:
        data = {
            "data_category": target,
            "rule_id": erasure_rule.id,
        }
        compound_key = to_snake_case(RuleTarget.get_compound_key(data=data))
        data["key"] = compound_key
        try:
            RuleTarget.create(
                db=db_session,
                data=data,
            )
        except KeyOrNameAlreadyExists:
            # This rule target already exists against the Policy
            pass

[ThomasLaPiana]

looking at previous tests runs, I definitely broke this. I think I know what I did but will confirm and fix

[ThomasLaPiana]

server issues sorted! now fixing up the tests

[ThomasLaPiana]

@seanpreston can you give this a review now? classic switcheroo

[ThomasLaPiana]

I'm seeing intermittent issues here when resetting the database 🤔 this needs further investigation and should not be merged until it is fixed

This is where it's hanging:

2022-10-13 13:16:41.291 [INFO] (main:log_request:262): Request received | {'method': 'GET', 'status_code': 200, 'handler_time': '513.392ms', 'path': '/health'}
2022-10-13 13:17:03.135 [INFO] (database:reset_db:64): Resetting database...

I have a sneaking suspicion that the database drop is colliding with another connection

[ThomasLaPiana]

ok, I've added some changes in that should make this more safe, and it wasn't triggered during the CI run but I'm also not 100% if its been fixed. Will warn the engineering team and keep an eye on it

[NevilleS]

I pushed some changes here and added some comments, but otherwise LGTM 👍

[ThomasLaPiana]

I noticed the hanging test is still happening intermittently, so I added a timeout for all tests of 15 minutes