Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

1357 celery on failure bindings #1411

Closed
wants to merge 584 commits into from

Conversation

eastandwestwind
Copy link
Contributor

Purpose

Convert email dispatch calls to async where possible.

Changes

Checklist

  • Update CHANGELOG.md file
    • Merge in main so the most recent CHANGELOG.md file is being appended to
    • Add description within the Unreleased section in an appropriate category. Add a new category from the list at the top of the file if the needed one isn't already there.
    • Add a link to this PR at the end of the description with the PR number as the text. example: #1
  • Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.
  • If docs updated (select one):
    • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
    • documentation issue created (tag docs-team to complete issue separately)
  • Good unit test/integration test coverage
  • This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
  • The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Ticket

Fixes #1357

Sean Preston and others added 30 commits July 7, 2022 11:35
* update celery config defaults, set to redis settings if none provided

* rename REDIS_CONNECTION_URL to CONNECTION_URL

* add var to config reference

* make DB index optional, use f'string

* adds support for redis USER

* add default for db_index in format string

* set correct default for USER
Bumps [next-auth](https://github.com/nextauthjs/next-auth) from 4.5.0 to 4.9.0.
- [Release notes](https://github.com/nextauthjs/next-auth/releases)
- [Changelog](https://github.com/nextauthjs/next-auth/blob/main/CHANGELOG.md)
- [Commits](https://github.com/nextauthjs/next-auth/compare/[email protected]@v4.9.0)

---
updated-dependencies:
- dependency-name: next-auth
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Handle hashed password

* Update CHANGELOG

* Fix failing test and use UserPasswordReset schema from fideslib

* Restore fidesops.toml

Co-authored-by: Paul Sanders <[email protected]>
* update local deployment for serving the ui

* changelog
* Update get_fides_log_record_factory to skip masking of all uvicorn logs.

* Update changelog.
…833)

* 832 - [User Management] Create new user gives HTTP 422 Unprocessable Entity exception

* Updated CHANGELOG.md file
* sign in to instead of sign into

Changing the wording of the admin ui page to improve grammatical accuracy.

* fides admin-ui wording changelog

* added link to pr

* fix typo on subject identities

* updated changelog

* typo
Co-authored-by: Paul Sanders <[email protected]>
Co-authored-by: eastandwestwind <[email protected]>
* replace clipboard icon

* temp fix for auth

* revert last change

* changelog
* Reduce docker image size

* Update CHANGELOG

Co-authored-by: Paul Sanders <[email protected]>
* adds identity fields to PrivacyRequest model

* store identity data inside database

* update changelog

* add identities in test data command

* store identities provided via the DRP creation endpoint

* black + isort

* store provided identity data in request creation from onetrust

* remove deprecated migration

* adds new provided identity table

* use new provided identity table

* add docstring, remove comment

* update DRP privacy request creation to use ProvidedIdentity model

* update identity creation in test data command

* use persisted identity in OneTrust

* update test to use persisted identity

* isort update

* use enums

* optionally receive a salt in hash_value cmd

* use a constant salt for provided identity hashing

* remove import

* use typehints

* update typedef

* use enum in dict
Bumps [faker](https://github.com/joke2k/faker) from 13.14.0 to 13.15.0.
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](joke2k/faker@v13.14.0...v13.15.0)

---
updated-dependencies:
- dependency-name: faker
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [unidecode](https://github.com/kmike/text-unidecode) from 1.2.0 to 1.3.4.
- [Release notes](https://github.com/kmike/text-unidecode/releases)
- [Commits](https://github.com/kmike/text-unidecode/commits)

---
updated-dependencies:
- dependency-name: unidecode
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on [fastapi-pagination[sqlalchemy]](https://github.com/uriyyo/fastapi-pagination) to permit the latest version.
- [Release notes](https://github.com/uriyyo/fastapi-pagination/releases)
- [Commits](uriyyo/fastapi-pagination@0.8.3...0.9.3)

---
updated-dependencies:
- dependency-name: fastapi-pagination[sqlalchemy]
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [black](https://github.com/psf/black) from 22.3.0 to 22.6.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](psf/black@22.3.0...22.6.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Refactor pages to use the same Layout

* Add UserForm.tsx

* Add Yup validation schema

* Integrate refactored form to New route

* Styling

* WIP refactor EditUserForm

* Fix EditUserForm typescript errors

* Refactor handleSubmit

* Small fixes to EditUserForm

* Make call to get scopes earlier so form will be populated on first render

* Update changelog
* adds identity fields to PrivacyRequest model

* store identity data inside database

* update changelog

* add identities in test data command

* store identities provided via the DRP creation endpoint

* black + isort

* store provided identity data in request creation from onetrust

* remove deprecated migration

* adds new provided identity table

* use new provided identity table

* add docstring, remove comment

* update DRP privacy request creation to use ProvidedIdentity model

* update identity creation in test data command

* use persisted identity in OneTrust

* update test to use persisted identity

* isort update

* use enums

* optionally receive a salt in hash_value cmd

* use a constant salt for provided identity hashing

* remove import

* use typehints

* update typedef

* use enum in dict

* test for exact match search

* added exact match search to request status api

* import order

* update CHANGELOG
* copy changes to fix docs outlining [#427] and standardize formatting [DOX-216]
* Fix create_test_data

* Update CHANGELOG

Co-authored-by: Paul Sanders <[email protected]>
Co-authored-by: Sean Preston <[email protected]>
Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.9.3 to 2.20.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v2.9.3...v2.20.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…FE) (#845)

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

* Updated CHANGELOG.md file

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Resolved import statement issues

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

1. Updated connector svg images
2. Added default style to Spinner component

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Deleted unnecessary file

* 671 - [Datastore Management] Include Icon with datastore connection (FE)

Updated MySQL and Outreach .svg files

* Renamed connectors folder to connector-logos

* Resolved misspelling error in import statement

* Code review feedback

* Code review feedback
* adds option to configure EVENT_QUEUE_PREFIX for celery

* provide the option to specify a default queue name too

* update celery config to load in from its own config toml file

* updates changelog

* update value for event_queue_prefix

* test celery config overrides

* include config_path arg

* add type def

* add config path to execution settings

* correct values
* update config reference

* additional documentation on celery configs [#755]

* add celery.toml reference

* define optional celery config and overrides

* reword config links

* link to lowercase settings
* updates changelog

* update ERD

* use star
Paul Sanders and others added 27 commits September 15, 2022 12:31
* Rename PrivacyRequestIdentity schema to Identity

* Update changelog

Co-authored-by: Paul Sanders <[email protected]>
* Make a manual webhook "testable" - which just asserts that a valid manual webhook configuration exists.

* Allow dsr_package_label to be an empty string but then convert to match pii field in that case.

* Changelog

* Respond to CR

* Fix mypy error.
* Explicitly install `toml` as a requirement

Previously this package was included as an indirect dependency only, but
imported in source code as if it were a direct dependency. This led to
installation issues for consumers of the `fidesops` `pip` package.

* Reduce `toml` import volume

* Update `CHANGELOG.md`
* Add consent crequest table

* Update changelog

* Fix isort and black

Co-authored-by: Paul Sanders <[email protected]>
…iguration (#1247)

* Added Saas dataset configuration

* Edits to Saas dataset configuration

* Edits to Saas dataset configuration

* Edits to Saas dataset configuration

* Refactor Connector Parameters

* Refactor Connector Parameters

* Refactor Connector Parameters

* Refactor Connector Parameters

* Reset Connection state when AddConnection component is unmounted

* Added Connection Identifier validation

* Updated ConnectionIdentifier validation to be removed from the JSX markup inline

* Added the verify=false to the query string for the API route to update the connection config secrets. This is to not connect directly to a given DB connector.

* Refactored Connector Parameters

* Reset Connection object state when the ConfigureConnector component is mounted

* Updated Connector Parameters

* Updated the toast notification placement to be positioned via top

* Updated CHANGELOG.md file

922 - Add a Connector - DB connector configs
923 - Add a Connector - upload a DB Dataset YAML
1090 - Add a Connector - SaaS Dataset Management (YAML method)

* Updated CHANGELOG.md file

* Resolved Prettier format issue

* Resolved npm run build issues

* add permissions to list

* Refactored Dataset Configuration

* Refactored Dataset Configuration

* Refactored Dataset Configuration

* Refactored Dataset Configuration

* 1015 - Frontend - Configure a Manual entry connector

* 1015 - Frontend - Configure a Manual entry connector

* 1015 - Frontend - Configure a Manual entry connector

* 1015 - Frontend - Configure a Manual entry connector

* Updated the UI to display the human readable value for the selected Connector

* Code review feedback changes

* Remove commented out code which is unnecessary

* Redirect the user back to the Datastore Connections landing page

* Updated DSR Customization user instruction verbiage

* Removed hard coded string literals in ConfigureConnector component

* If a connection has been initially created, then auto redirect the user accordingly

* Resolved issue with ConnectionsEmptyState component not be displayed when there are no connections

* Reverted fidesops.toml file

* add scopes to allowed list of permissions

Co-authored-by: Sean Preston <[email protected]>
* pin version of pyjwt

* assert against encoded identity not hardcoded
* Updating assertions and using a unique erasure identity email per test run

* Fixing sort order
#1349)

* Fix bug where erasure count confirmations may be returned instead of selected access request results when attempting to get access requests from the cache.

Because erasure requests are run after access requests, there normally isn't erasure results in the cache when we attempt to get access results.  However, if we were retrying a privacy request from a checkpoint that had already erased data, we could possibly return erasure results instead of access results, depending on if the access or erasure key was accessed last.

* Update CHANGELOG.
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.1.2 to 7.1.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@7.1.2...7.1.3)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [types-ujson](https://github.com/python/typeshed) from 5.4.0 to 5.5.0.
- [Release notes](https://github.com/python/typeshed/releases)
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-ujson
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* updates changelog

* updates ERD

* add an execution_timeframe to quickstart policies

* turn feature flag on

* format flags
* For UI purposes, when retrieving any manually input data, distinguish whether the user has confirmed they looked for data, in that they've manually uploaded something, even if everything was empty. In this case: checked=True.

* Prevent entering a pii or dsr_label as "      ".

* Fix unit test.

* Update Changelog.

Co-authored-by: Sean Preston <[email protected]>
* 1310- use existing celery task for all email dispatch

* changelog

* force sync for some use cases, update tests accordingly

* format

* fix integration tests
* Reset the cache expiration in testing so it doesn't potentially affect other long-running tests that depend on the cache.

* Update Changelog.
* Add api_key authentication strategy

* Update SaaS configs to use new api_key auth strategy

* Update changelog

* Simplify api_key auth config by requiring lists for fields

Update field names to be plural to reflect that they are multi-value lists/arrays.

* Remove query_param auth strategy

The query_param auth strategy can now easily be implemented as an
`api_key` auth strategy, which provides the same functionality and more.

* Update api_key auth model to reference Header model class

* Update changelog to include query_param auth strategy removal note

Co-authored-by: Adam Sachs <[email protected]>
@eastandwestwind eastandwestwind deleted the 1357-celery-on-failure-bindings branch September 30, 2022 17:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use on_failure bindings for Celery email tasks that require special handling on failure to send