Proposal for Quota Reporting and Management

enhancements/quota-management.md

@@ -0,0 +1,99 @@
+---
+title: Quota Management and Enforcement on Quay
+authors:
+  - "@kwestpha"
+  - "@sleesinc"
+  - "@sdadi"
+reviewers:
+  - "@kwestpha"
+  - "@sleesinc"
+  - "@sdadi"
+approvers:
+  - "@kwestpha"
+  - "@sleesinc"
+  - "@sdadi"
+creation-date: 2021-12-03
+last-updated: 2021-12-03
+status: implementable
+---
+
+# quota-management-and-enforcement
+
+Currently, Quay users have very little control to contain unbounded growth of storage consumption of the registry.
+This proposal is to report storage consumption and contain registry growth based on the configured storage quota limits.  
+
+## Release Signoff Checklist
+
+- [x] Enhancement is `implementable`
+- [x] Design details are appropriately documented from clear requirements
+- [ ] Test plan is defined
+- [ ] Graduation criteria for dev preview, tech preview, GA
+
+## Open Questions
+
+ > 1. Do we have any here?
+
+## Summary
+
+There are no limitations on a Quay organization in terms of their storage consumption. This is a three-phase proposal that
+aims to contain and provide a smart solution to tackle storage related problems for Quay customers.   
+In the first phase, a super admin can see the storage consumption of all of their organizations.
+While a user can see the storage consumption of their organization. In the second phase, a superuser can define soft and hard checks for Quay users.
+A soft check implies that the Quay user will be notified if the storage consumption of an organization reaches the configured threshold.
+A hard check implies that the Quay user will not be allowed to push to the registry once the storage consumption reaches the set limit.
+In the third phase, a quay admin can configure auto pruning of the images based on when they were last used.    
+
+## Motivation
+
+Users can create new organizations and repositories in their home organizations without any practical storage limitation.
+On-prem Quay customers are subject to capacity limits of their environment, specifically backend storage consumption.
+No knowledge about their storage consumption makes it hard for a service owner of a Quay registry to define SLAs and maintain a certain budget.
+and containing registry growth and utilization in terms of storage consumption.
+
+
+### Goals
+
+* A Quay superuser can see the overall storage consumption of the entire registry i.e., an overview of all organizations depicting the aggregate storage consumption of images in an organization.
+* A Quay user can define soft quotas (storage limit) on an organization that will trigger a warning when the set limit is reached.
+* A Quay user can configure alerting thresholds on the storage quota that will allow Quay to send notifications when set threshold is reached.
+* A Quay user can define hard quotas (storage limit) on an organization which will not allow them to push any more images when set threshold is reached.
+
+### Non-Goals
+
+??
+
+## Design Details
+
+Initial work is to add Quota Reporting on organizations i.e., providing Quay users with their storage usage on a Quay organization.
+In the next steps, we will add Quota Enforcements i.e., soft and hard checks that notify or restrict Quay users from pushing images to registry based.
+
+The expected design flow is depicted as below:
+![](https://user-images.githubusercontent.com/11522230/144654849-62d00b53-3be7-4009-afb6-2523f2931841.jpeg)
+NOTE: The boxes in black show the current flow, and the green boxes show what needs to be implemented in this feature. Design credits: @kwestpha
+
+The table RepositorySize (to be created) stores the storage consumption(in KBs) of a Quay repository within an organization. The sum of all repository sizes for an organization,
+defines the current storage size of a Quay organization.
+When an image push is initialized, the Quay users organization storage is validated to check if it is beyond the configured Quota limits.
+If yes, for a soft check, an exception is raised and, for a hard check, the push is stopped. If no i.e, storage consumption is within configured Quota limits,
+the push happens as expected.
+
+Images are uploaded to the Quay registry as chunks. Each of these chunk upload triggers, the next flow.
+The size of each image chunk is stored a temporary table, TemporaryUsage (to be created) that has a foreign key reference to the earlier mentioned RepositorySize table.
+This size of every chunk upload is added to the current repository size that calculates a temporary size. This temporary size is validated to be
+within the configured Quota limits. If this temporary size reaches beyond the configured Quota limit, an exception is raised.
+
+TODO: Add design flow of images deletions from repositories
+
+### Risks and Mitigations
+
+* TODO: @kwestpha to add the discussed risk.
+
+### Test Plan
+
+* Implement Quota Reporting on organization level.
+* Implement Quota Management for soft and hard checks on an organization level.
+* TBD: Implement Auto Pruning
+
+### Implementation History
+
+* 2021-12-03 Initial proposal