Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store): Migration overhaul (bun orm) #396

Merged
merged 31 commits into from
Nov 17, 2023
Merged

Conversation

jay-dee7
Copy link
Member

@jay-dee7 jay-dee7 commented Aug 11, 2023

Motivation & Context:

This is a gigantic effort to refresh OpenRegistry & redo some of the pending restructuring. The persistence layer was tightly coupled with a database & extending it was difficult.

Description:

We add new features, re-structure some old ones, make existing persistence layer flexible by using an ORM (bun) and update our implementation of OCI specific APIs.

OCI Features and Changes

  • We add the new Referrers API - OCI Spec Link
  • Use the official data structures for storing OCI descriptors (image manifest, manifest config, layers, etc all use the Image Spec's Descriptor type as recommended by the OCI team)
  • We now store the Image manifest locally instead of uploading to enabled DFS. Manifests are usually small in size & are in JSON format, which makes storing them easier & operations on them become a lot easier.
  • Use OCI Dist Spec's github.com/opencontainers/go-digest/Digest type for working with digests
  • Push Manifest API transparently accepts manifests with subject field
    • Push Manifest also creates a repository first if one does not exist

Registry Backend Features and Changes

  • Repositories can now we Private or Public
  • New Create Repository API, which defaults to private visibility
  • New Public Catalog API, which returns a public catalog to repositories (all public repositories)
  • New Get User Catalog API, which returns a user's catalog, which can be filtered by public or private visibility modes
  • New Change Repository Visibility API, which let's a user toggle the visibility of a repository
  • New Repository Propagation Middleware, which sets the requested repository in request's context on all repository namespaced endpoints

Persistence Store Changes

Logging & Telemetry

  • We've added OpenTelemetry based metrics
  • Fixed Logging & log formats
  • Added axiom.co as a possible log destination

Operational improvements

  • Built-in store/database migration system backend by The Uptrace Bun ORM
  • New command line interface
    • migrations - Run & manage database migrations
      • init - Initialise the database with all the tables, users, permissions, etc
      • run - Run database migrations
      • rollback - Rollback the latest migration
      • generate - Generate a new migration file
      • reset - Recreate the database by deleting it first
    • start - Start the OpenRegistry server
    • extras - Helpful commands like creating digest of random input

Types of Changes:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

PR Checklist:

  • My code follows the code style of this project
  • My change requires a change to the documentation
  • I have updated the documentation accordingly
  • I've read the CONTRIBUTION guide
  • I have signed-off my commits with git commit -s

@jay-dee7 jay-dee7 self-assigned this Aug 11, 2023
@jay-dee7 jay-dee7 force-pushed the private-container-images branch 23 times, most recently from f93cdba to 57e1d8e Compare August 14, 2023 15:29
@jay-dee7 jay-dee7 added this to the IPFS Milestone 2 milestone Aug 31, 2023
@jay-dee7 jay-dee7 linked an issue Aug 31, 2023 that may be closed by this pull request
@jay-dee7 jay-dee7 marked this pull request as ready for review November 17, 2023 13:58
@guacamole guacamole merged commit 2494e9e into main Nov 17, 2023
8 checks passed
@guacamole guacamole deleted the private-container-images branch November 17, 2023 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request github_actions Pull requests that update GitHub Actions code P1 P1 label is used for this that we see of highest priority. Critical bugs, security issues, etc
Projects
Status: Done
2 participants