Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STORY: Server supports user-specific data set retention and expiration thresholds #2019

Open
dbutenhof opened this issue Dec 3, 2020 · 0 comments
Labels
API Of and relating to application programming interfaces to services and functions Backlog enhancement Server

Comments

@dbutenhof
Copy link
Member

dbutenhof commented Dec 3, 2020

The user profile includes a "retention policy" so that extremely old data sets are automatically removed. The dashboard mocks show a display of data sets that are going to expire.

This epic can be broken down into three separate stories, each of which may be represented by one or more development tasks (e.g., database support vs API support):

  1. Backend implementation of a mechanism to store, update, and retrieve user-specific retention (and warning?) periods.
  2. Backend implementation of a mechanism to store, update, and retrieve server default retention/warning periods.
  3. Frontend mechanism for user to view and set their own retention fields, showing the limitations established for the server (e.g., enter number of days to retain a dataset, which can't be more than nn days as set by server administrator)
  4. The mechanism by which datasets are expired within the server.
  5. Optionally, a frontend mechanism for a server ADMIN to view and set the server default values

Ideally, these mechanisms will not be limited specifically to dataset retention properties, but can be arbitrarily expanded to handle additional user- and server- configuration properties. (E.g., #2049)

This issue deals with the user-specific implementation for

  1. retention period (age at which a dataset is deleted)
  2. warning threshold (amount of warning before deletion)

These properties become part of the User profile. TBD whether they are added as native fields in the User sqlalchemy model, or implemented as user Metadata records.

We need to implement server backend API(s) and infrastructure to enable this. There are some questions to be resolved, but here's a possible workflow:

  1. The server provides a default retention period, possibly a maximum retention period, and a default warning period;
  2. When a user is created, the user profile inherits the server default retention period and warning period;
  3. A user can alter their own retention period within the maximum retention period range;
  4. A user can alter their own warning period from 0 (disabled) to their retention period;
  5. A backend service periodically deletes old data sets based on the owning user and that user's retention period;
  6. There is a server API query probably taking as input only a "user" name property, which will return a list of all data sets owned by that user that are within the user's warning threshold of the user's retention period. (E.g., if the retention period is 90 days and the warning threshold is 30 days, all data sets that are 60 to 90 days old would be returned.) The dashboard will use this query on user login to populate the list of "due to expire" data sets.

This implies

  1. Implementation of a user profile (presumably extended PostgreSQL tables linked to the primary user record);
  2. Obviously we'll need to understand the ownership of data sets.
  3. We need to be able to identify all data sets owned by the user within the horizon (e.g., using Dataset table)

This API could be prototyped for experimentation as a query for controller names with an age between arbitrary warning period A and arbitrary expiration period B (e.g., 60 to 90 days).

Related to #2048

@portante portante added this to the v0.72 milestone Dec 3, 2020
@portante portante removed the Backlog label Jun 24, 2021
@dbutenhof dbutenhof self-assigned this Jul 30, 2021
@dbutenhof dbutenhof added API Of and relating to application programming interfaces to services and functions and removed help wanted labels Jul 30, 2021
@dbutenhof dbutenhof removed their assignment Jul 30, 2021
@dbutenhof dbutenhof changed the title Server API/infrastructure to implement user data set retention and expiration warnings STORY: Server supports user-specific data set retention and expiration thresholds Jul 30, 2021
@portante portante removed this from the v0.72 milestone Sep 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Of and relating to application programming interfaces to services and functions Backlog enhancement Server
Projects
None yet
Development

No branches or pull requests

4 participants