Skip to content
This repository has been archived by the owner on Nov 30, 2022. It is now read-only.

Add a Timescale Connector [#1311] #1327

Merged
merged 12 commits into from
Sep 23, 2022
Merged

Conversation

pattisdr
Copy link
Contributor

@pattisdr pattisdr commented Sep 15, 2022

❗ Contains a migration; verify downrev before merging

Purpose

Add a connector for Timescale and demonstrate that it can make the standard access and erasure requests.

Changes

Added a Timescale connector that is built entirely off of our existing PostgreSQL connector. This has the same limitations of our current PostgreSQL connector in that we don't yet support complex types, among other things.

The code added here is largely boilerplate very similar to our postgres connector. We should explore ways in the future to create factories that could populate similar information for different connectors. This was a lot of copy/paste/adjusting the relevant variables. The main things that are different are a slightly different docker-compose and the introduction of the hypertable construct.

Checklist

  • Update CHANGELOG.md file
    • Merge in main so the most recent CHANGELOG.md file is being appended to
    • Add description within the Unreleased section in an appropriate category. Add a new category from the list at the top of the file if the needed one isn't already there.
    • Add a link to this PR at the end of the description with the PR number as the text. example: #1
  • Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.
  • If docs updated (select one):
    • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
    • documentation issue created (tag docs-team to complete issue separately)
  • Good unit test/integration test coverage
  • This PR contains a DB migration. If checked, the reviewer should confirm with the author that the down_revision correctly references the previous migration before merging
  • The Run Unsafe PR Checks label has been applied, and checks have passed, if this PR touches any external services

Ticket

Fixes #1311

@pattisdr pattisdr changed the title Timescale POC [#1311] Add a Timescale Connector [#1311] Sep 20, 2022
('[email protected]', '2022-01-07 09:00:00'),
('[email protected]', '2022-01-08 09:00:00');

SELECT create_hypertable('onsite_personnel', 'time', migrate_data => true);
Copy link
Contributor Author

@pattisdr pattisdr Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Creates a hypertable here by first creating a postgres table and then converting it into a hypertable. You have to specify the column that holds the "time" values. In this contrived example, I have unique times that should be the primary key with a "responsible" employee who was on call at that time.

I've added an integration test that demonstrates a basic access and erasure request on this hypertable. Some comments are that we require primary keys to erase. Noting that timescale requires the unique/primary key dimension to contain the time dimension. You can have composite primary keys including the time dimension in Timescale and fidesops has support for this.

Also worth noting that "datetimes" are not a fully supported date type in Fidesops yet. A lot of our existing examples that deal with dates tend to treat dates as strings. Masking date fields can be problematic outside the null masking strategy. (Although since the date field is going to be part of the primary key, it is less likely that this is a field that is masked by itself).

@pattisdr pattisdr marked this pull request as ready for review September 21, 2022 18:33
@pattisdr pattisdr force-pushed the fidesops_1311_timescaledb_poc branch from 3d2f3b3 to c98e5d7 Compare September 22, 2022 18:49
@sanders41 sanders41 merged commit ebd9e7d into main Sep 23, 2022
@sanders41 sanders41 deleted the fidesops_1311_timescaledb_poc branch September 23, 2022 14:01
@pattisdr
Copy link
Contributor Author

Thanks very much for your comments @sanders41!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Spike] TimescaleDB
2 participants