Fidesops merge to unified fides 2 #1214

chriscalhoun1974 · 2022-10-04T14:52:29Z

Closes #1215

Code Changes

Merge fidesops/main into unified-fides-2 git branch

get the server up and running
get the tests collecting & running
get the frontend up and running
frontend tests are working

Steps to Confirm

automated tests pass (all that can, doesn't include mypy)
server spins up and runs
UI is usable

Pre-Merge Checklist

All CI Pipelines Succeeded
Documentation Updated:
- documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
- documentation issue created (tag docs-team to complete issue separately)
Issue Requirements are Met
Relevant Follow-Up Issues Created
Update CHANGELOG.md

Description Of Changes

Merging fidesops main into unified-fides, as of version 1.8.1 of fidesops

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.9.3 to 2.20.0. - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v2.9.3...v2.20.0) --- updated-dependencies: - dependency-name: pre-commit dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ss subject request (#1377) * On successful connector creation the empty yaml text input displays a validation error * Updated caniuse-lite NPM dependency * Added min/max character limitation validation for Manual Webhook DSR Customization * Prevented Chakra Divider component opacity from being overridden at runtime. User could not see visible horizontal line on UI before. * Updated the Manual Processing detail form to not be submitted until the form is dirty. * Removed form dirty check on ManualProcessingDetail component * Update URL when user is navigating to DSR Customization screen when creating a Manual Webhook

…vel (#1376) * Add decorator to send method that retries throttles * Fix black/mypy * Fix pylint * Add tests for authenticated client * Small fixes and typos * Update CHANGELOG.md * Small changes. Update retry logic to not retry general exceptions Co-authored-by: Eduardo Armendariz <[email protected]>

Co-authored-by: Paul Sanders <[email protected]>

* Add new template for consent requets emails * Remove analytics id * Update subject Co-authored-by: Paul Sanders <[email protected]>

Co-authored-by: Paul Sanders <[email protected]>

* If a manual webhook is deleted or disabled, check if there are any remaining active manual webhooks configured. If not, queue any Privacy Requests stuck in "requires_input" for processing. * In the "view_uploaded_manual_webhook_data", load cached webhook data for a privacy request in strict mode. If it fails (no data saved, extra field saved, field missing), return checked=True, so the user knows they need to reupload data for this webhook before it can be submitted. Return the data in non-strict mode, so we just show the overlap between the data saved and the fields defined. * Update changelog. * Move queue_requires_input_requests to the connection_endpoints where this is the only module it's being called - both where you update and delete a connection config. * Clarify docstring.

) * add sane defaults * make subsections of config with complete defaults optional * lowercase database.enabled, set defaults for optional configs * update return type * updates changelog * make PORT an env var * cast env var to int * remove unnecessary unpinned dependency * bump fideslib version * bump fideslib to 3.1.4 * add defaults for the non optional config subclasses * set empty dict to default for config subclasses that require some fields * use .get() in assemble URL for correct error message, correct comment * update jwt_key type annotation

Co-authored-by: Cole <[email protected]>

removing the typo ` . at the end of the command so that the command works.

* Add consent UI back in * Finish initial integration with consent api * WIP consent page * Get initial consent updating working * Improve button look and feel * Add untracked files * Format VerificationForm.tsx * Remove comments * Rename Privacy modal variables * Rename variable

* 1401 - Admin UI: Persist Redux store to localStorage * Resolved React memory leak when user attempts to logout via the Subject Requests landing page * Resolved UI unit test failure * Rollback previous change * Removed blacklist property from Redux store config * Refactored due to recommended code review feedback

* update footer links * changelog * Update CHANGELOG.md Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Sean Preston <[email protected]> Co-authored-by: Paul Sanders <[email protected]>

* Initial implementation of firebase auth connector Introduces firebase_admin sdk as a fidesops dependency Leverages request overrides to define custom functionality * Update changelog * Minimize unused client config in firebase auth config * Add user.photo_url as a supported Firebase Auth field Improve readability in access implementation * Fix field name to be more consistent with python SDK properties * Add clarifying comment to unused function docstring * Add photo_url field to firebase auth dataset * Properly test provider data in firebase auth integration. Also include test coverage for delete function, even thought it's not being invoked by default saas config. * Only add fields to result if popluated * Fix empty attribute logic within provider data block * Ignore pylint error for too many variables * fix up provider data check Co-authored-by: Adam Sachs <[email protected]>

* Braze SVG added * Fixing change log message Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: Adam Sachs <[email protected]>

* updates changelog * add UI testing steps to release checklist

* stop swallowing integrity errors on dataset update * dont attach logging middleware if analytics opt_out is true * handle diff correctly * disable consider-using-f-string as we need this for logging statements to work effectively in fideslog

…nified-fides-2

seanpreston · 2022-10-04T16:03:00Z

src/fides/api/ops/service/connectors/saas/authenticated_client.py

+
+        return decorator
+
+    @retry_send(retry_count=3, backoff_factor=1.0)  # pylint: disable=E1124


Hey @Kelsey-Ethyca, we have similar systems for retrying nodes in the traversal albeit with one major difference, that we're able to set the values with these config. variables. This comes with the added benefit of being able to moderate the values during testing — ie. the tests don't waste time waiting around for requests to retry that are always destined to fail.

Can we fast-track a change for this please?

ThomasLaPiana · 2022-10-04T17:39:48Z

got the webserver up and running! now going to update the tests where needed

ThomasLaPiana · 2022-10-04T18:12:39Z

@chriscalhoun1974 @TheAndrewJackson what's the status of the frontend at this point?

ThomasLaPiana · 2022-10-04T18:36:47Z

going to add the new dataset tables now to get fides checks passing

TheAndrewJackson · 2022-10-04T19:01:54Z

@ThomasLaPiana The frontends have been merged. Now we're going through and making sure everything runs/builds.

ThomasLaPiana · 2022-10-04T19:05:58Z

awesome, I'm just about done fixing up the backend and tests

TheAndrewJackson · 2022-10-04T19:41:56Z

@ThomasLaPiana I got everything running. Now we just need to fix 2 failings tests and do a once through on the UI to make sure nothing weird snuck in and we should be good!

…m/ethyca/fides into fidesops-merge-to-unified-fides-2

ThomasLaPiana · 2022-10-04T19:48:28Z

@TheAndrewJackson excellent! ping me when its good to go and I'll merge once I'm done with the backend too! Thanks so much 🙏

ThomasLaPiana · 2022-10-04T20:16:49Z

seeing lots of green! going to fix these conflicts

…m/ethyca/fides into fidesops-merge-to-unified-fides-2

TheAndrewJackson · 2022-10-04T21:01:41Z

@ThomasLaPiana Just confirmed with Chris that the UI passed his smoke test and that all of the tests are passing. Everything looks good on the UI side

ThomasLaPiana · 2022-10-04T21:29:32Z

all expected checks passing, shipping it

dependabot bot and others added 21 commits September 27, 2022 09:43

Add Consent Request API (#1387)

406d3c8

Co-authored-by: Paul Sanders <[email protected]>

Add new template for consent requets emails (#1405)

6879403

* Add new template for consent requets emails * Remove analytics id * Update subject Co-authored-by: Paul Sanders <[email protected]>

Add authenticated route to get consent preferences (#1402)

23cdaa5

Co-authored-by: Paul Sanders <[email protected]>

Braze Connector: Access Endpoints (#1248)

e4a5816

Explain Privacy Request Execution [#1397] (#1396)

060152b

Co-authored-by: Cole <[email protected]>

Update docker command on privacy center step 4 (#1410)

13fabfe

removing the typo ` . at the end of the command so that the command works.

update footer links (#1406)

b966074

* update footer links * changelog * Update CHANGELOG.md Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Sean Preston <[email protected]> Co-authored-by: Paul Sanders <[email protected]>

Add Braze connector registry entry (#1418)

b42e6d7

* Braze SVG added * Fixing change log message Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: Adam Sachs <[email protected]>

Removing PII fields from conversations collection (#1386)

59cd938

Fidesops v1.8.1 Release Checklist (#1420)

3c30339

* updates changelog * add UI testing steps to release checklist

Merge remote-tracking branch 'fidesops/main' into fidesops-merge-to-u…

4c05f63

…nified-fides-2

Updated CHANGELOG.md file from prior merge

f5e7c82

chriscalhoun1974 requested a review from a team October 4, 2022 14:52

chriscalhoun1974 marked this pull request as draft October 4, 2022 14:52

chriscalhoun1974 marked this pull request as ready for review October 4, 2022 15:18

seanpreston reviewed Oct 4, 2022

View reviewed changes

ThomasLaPiana assigned TheAndrewJackson and chriscalhoun1974 Oct 4, 2022

ThomasLaPiana marked this pull request as draft October 4, 2022 16:56

get the server up and running

8ee6849

fix test collection

46c17b4

fix isort and black

1842df4

update dataset annotations

291e9fc

TheAndrewJackson added 3 commits October 4, 2022 15:32

Merge latest fidesops UI into unified ui

77bf8d8

Fixing post UI merge issues

cb4df3b

Fix formatting and import issues

babef34

ThomasLaPiana added 2 commits October 4, 2022 12:46

get all unit tests passing

82a1c6e

Merge branch 'fidesops-merge-to-unified-fides-2' of https://github.co…

ac94dbd

…m/ethyca/fides into fidesops-merge-to-unified-fides-2

Fix cypress failure 🤞

fdb7fe6

TheAndrewJackson and others added 3 commits October 4, 2022 16:33

Remove auth.slice.test.ts

df921ab

Merge branch 'unified-fides-2' into fidesops-merge-to-unified-fides-2

890abcb

Merge branch 'fidesops-merge-to-unified-fides-2' of https://github.co…

331d4ec

…m/ethyca/fides into fidesops-merge-to-unified-fides-2

ThomasLaPiana marked this pull request as ready for review October 4, 2022 21:30

ThomasLaPiana merged commit 2463c82 into unified-fides-2 Oct 4, 2022

ThomasLaPiana deleted the fidesops-merge-to-unified-fides-2 branch October 4, 2022 21:30

SteveDMurphy mentioned this pull request Oct 5, 2022

[Unified Fides] Add Missing Dataset Annotations #1187

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fidesops merge to unified fides 2 #1214

Fidesops merge to unified fides 2 #1214

chriscalhoun1974 commented Oct 4, 2022 •

edited by ThomasLaPiana

Loading

seanpreston Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

TheAndrewJackson commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

TheAndrewJackson commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

TheAndrewJackson commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022


		return decorator

		@retry_send(retry_count=3, backoff_factor=1.0) # pylint: disable=E1124

Fidesops merge to unified fides 2 #1214

Fidesops merge to unified fides 2 #1214

Conversation

chriscalhoun1974 commented Oct 4, 2022 • edited by ThomasLaPiana Loading

Code Changes

Steps to Confirm

Pre-Merge Checklist

Description Of Changes

seanpreston Oct 4, 2022

Choose a reason for hiding this comment

ThomasLaPiana commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

TheAndrewJackson commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

TheAndrewJackson commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

TheAndrewJackson commented Oct 4, 2022

ThomasLaPiana commented Oct 4, 2022

chriscalhoun1974 commented Oct 4, 2022 •

edited by ThomasLaPiana

Loading