Unify ops/ctl web applications

[ThomasLaPiana]

Closes #1011

Code Changes

• do another merge of fidesops/main
• move all of the required ops_main.py logic into main.py
• fix policy & policy webhook endpoint clashes
• fix user duplicated endpoints
• dedupe health endpoint
• get all of the tests passing
• copy over all of the required secrets for GH actions
• deduplicate anything that makes sense to deduplicate related to server infra/configuration

Steps to Confirm

• tests pass 🚀

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation Updated:
  • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
  • documentation issue created (tag docs-team to complete issue separately)
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md

Description Of Changes

These changes are the lynchpin of the Unified Fides ™️ project, as this PR fully merges both applications into a single webserver. Further work will be done down the line to continue deduplicating certain code and cleaning things up, however with this change ctl & ops will share a single webserver + database

This PR makes some choices around which pieces to take from which server implementation for merging into a single main.py file, so any reviews should mostly focus on the main.py changes

[ThomasLaPiana]

the final test errors here are in the request_runner_service_test.py file, not sure what's breaking here so I'm going through and trying to carefully discern why some tests in that file are passing while others are failing

[ThomasLaPiana]

so looking at the test failures, it doesn't make much sense...I've printed out the values for the failing postgres test and they are as follows:

The customer table after deletion:
[(2, 'Jill Customer'), (3, 'Jane Customer'), (1, None)]
Looking for customer_id: 1
In row: (2, 'Jill Customer')
Looking for customer_id: 1
In row: (3, 'Jane Customer')
Looking for customer_id: 1
In row: (1, None)

However the expected match of 1 in (1, None) isn't happening oddly enough

As can be seen, the id absolutely exists in the row, but for some reason it isn't matching. Seems like good news as this seems to suggest there is no larger issue here other than maybe a small code fix in the test itself

[ThomasLaPiana]

I fixed the failing tests but I don't quite understand why they broke in the first place @sanders41

The solution was that the comparison wasn't working, but it seemed valid to me 🤔

This is what I used that works:
if customer_id == row.id:

This is what was there previously that didn't work but I'm not sure why:
if customer_id in row:

[ThomasLaPiana]

all non external/saas tests are passing!

next step is to copy over all of the secrets that ops needs

[ThomasLaPiana]

@sanders41 @seanpreston can you help me get the secrets copied over? I couldn't find them in 1pass under the same name, and those are the last tests left to check

[sanders41]

Very weird on the comprehension check. Every reason I can think of that would make if customer_id in row: fail would also make if customer_id == row.id: fail. Really if customer_id == row.id: is more strict so if it works lets go with it.

@eastandwestwind can you help with the secrets? I think these will be the vault secrets.

[eastandwestwind]

@ThomasLaPiana we shouldn't need to copy over vault secrets into GH secrets, since ops will only look in Vault for secrets at this point. Btw, you might not have proper access to view the fidesops secrets in 1pass (message me to get more details on which folder to look for), but @daveqnet could help you out with access.

[ThomasLaPiana]

@ThomasLaPiana we shouldn't need to copy over vault secrets into GH secrets, since ops will only look in Vault for secrets at this point. Btw, you might not have proper access to view the fidesops secrets in 1pass (message me to get more details on which folder to look for), but @daveqnet could help you out with access.

this would be for:

1. The vault secrets aren't exactly specified in 1pass, so I'm not sure what to add to the repo
2. the external integration tests (redshift, bigquery, snowflake) still appear to use non-vault secrets

[ThomasLaPiana]

State of the PR:

• Ops External Integration tests have 2 failing Snowflake tests
• Ops External SaaS tests have 2 failing Hubspot tests

I'm going to merge now, as these tests shouldn't hold up this PR or the rest of the work. They can be fixed in a smaller PR