feat: introducing oci artifact registry impl #2989

jonathanj-square · 2024-10-04T00:16:06Z

Introduces the OCI implementation of the module registry. The controller still uses the database module registry; so to exercise the change the ftl release commands have been introduced to demonstrate pushing module artifacts to the registry and discovering them (and metadata). The CLI implementations will change with the subsequent controller integration work.

- adding docker registry image

- integration with CLI to demonstrate release publishing is WIP

docker-compose.yml

alecthomas · 2024-10-09T22:34:28Z

backend/controller/artefacts/oci_registry.go

+}
+
+func (s *ContainerService) Upload(ctx context.Context, artefact Artefact) (sha256.SHA256, error) {
+	hash := sha256.Sum(artefact.Content)


Doesn't Artefact already have a sha256 sum?

right you are, using that instead

alecthomas · 2024-10-09T22:41:20Z

backend/controller/artefacts/oci_registry.go

+	}
+	registry.PlainHTTP = true
+	result := make([]ArtefactRepository, 0)
+	err = registry.Repositories(ctx, "", func(repos []string) error {


Is this iterating over all repositories? That won't be scalable will it? eg. in Cash's cloud there will be thousands if not hundreds of thousands of repositories.

I don't think this operation is actually part of the OCI standard either. The underlying implementation queries "%s://%s/v2/_catalog" which is not part of the OCI distribution spec.

Either way there should be a single configured repository where we upload these artifacts, and we should not rely on the discovery APIs.

Created an issue to track fully spec'ing out the approach.

alecthomas · 2024-10-09T22:42:20Z

backend/controller/artefacts/oci_registry.go

+	return fmt.Sprintf("%s/%s", host, createModuleRepositoryPathFromDigest(digest))
+}
+
+// getDigestFromModuleRepositoryPath extracts the digest from the module repository path; e.g. /ftl/modules/<digest>:latest


This needs to be /ftl/modules/<modul>/<digest> no?

Where is this path actually defined BTW, I don't see it anywhere.

See my comment above, but basically this should be a config option and not per module so we can re-use artifacts.

alecthomas · 2024-10-09T23:21:46Z

backend/controller/artefacts/oci_registry.go

+	ModuleArtifactPrefix = "ftl/modules/"
+)
+
+type ContainerConfig struct {


FYI the typical pattern for config is to tag them with Kong tags, and embed them in the CLI so they can be automatically passed through from the CLI.

alecthomas · 2024-10-09T23:27:03Z

frontend/cli/cmd_release.go

+}
+
+type releaseListCmd struct {
+	Registry             string `help:"Registry host:port" default:"127.0.0.1:5001"`


These all seem to be shared, just put them in releaseCmd and inject that into the Run() method as a parameter.

neat, updated

stuartwdouglas · 2024-10-10T01:14:17Z

backend/controller/artefacts/oci_registry.go

+	}
+	registry.PlainHTTP = true
+	result := make([]ArtefactRepository, 0)
+	err = registry.Repositories(ctx, "", func(repos []string) error {


I don't think this operation is actually part of the OCI standard either. The underlying implementation queries "%s://%s/v2/_catalog" which is not part of the OCI distribution spec.

Either way there should be a single configured repository where we upload these artifacts, and we should not rely on the discovery APIs.

stuartwdouglas · 2024-10-10T01:22:32Z

backend/controller/artefacts/oci_registry.go

+
+func (s *ContainerService) Upload(ctx context.Context, artefact Artefact) (sha256.SHA256, error) {
+	hash := sha256.Sum(artefact.Content)
+	ref := fmt.Sprintf("ftl/modules/%s", hash)


This should be a config option. Basically the general location format should probably be something like <host>/<configured-repo>:ftl-artifact-<digest>.

The design doc should probably be updated to cover exactly how these artifacts are stored in the registry in terms of the naming scheme.

agreed, I need to be more explicit about how release and module compilation artifacts (binaries, jars, schema DSLs) get mapped to OCI format concepts and their layout:

Created a tracking item here: #3077

stuartwdouglas · 2024-10-10T01:29:32Z

backend/controller/artefacts/oci_registry.go

+	for _, d := range digests {
+		set[d] = true
+	}
+	modules, err := s.DiscoverModuleArtefacts(ctx)


This should be explicitly checking the registry for each digest, not listing the full registry then iterating over the list. Ideally Oras should be sending a HEAD request to /v2/<name>/blobs/<digest> for each artifact to check.

Even then I am not sure if this is the right approach, we can still store in the database which artifact we know about, as for JVM applications even the HEAD approach will likely be slower than we would want.

We could still potentially use the HEAD approach if we don't know about the artifact, as it would allow us to re-use jars from other clusters.

Great call and thanks for catching it at this stage. Some rethinking is due here, this ticket will track that.
#3076

stuartwdouglas · 2024-10-10T01:32:07Z

backend/controller/artefacts/oci_registry.go

+	return fmt.Sprintf("%s/%s", host, createModuleRepositoryPathFromDigest(digest))
+}
+
+// getDigestFromModuleRepositoryPath extracts the digest from the module repository path; e.g. /ftl/modules/<digest>:latest


See my comment above, but basically this should be a config option and not per module so we can re-use artifacts.

stuartwdouglas · 2024-10-10T01:36:58Z

frontend/cli/cmd_release.go

+	MaxIdleDBConnections int    `help:"Maximum number of idle database connections." default:"20" env:"FTL_MAX_IDLE_DB_CONNECTIONS"`
+}
+
+func (d *releasePublishCmd) Run() error {


If we are going to do 'releases' to a container registry we will need to push some kind of manifest with the contents of the release, as releases are multiple files.

agreed, the transition to the OCI registries involves multiple steps, the first is to simply dump the payloads there instead of the database. from there it will be refined to take advantage of the container formats capabilities

stuartwdouglas · 2024-10-10T01:38:23Z

frontend/cli/cmd_release.go

+	if err != nil {
+		return fmt.Errorf("failed to create container service: %w", err)
+	}
+	modules, err := svc.DiscoverModuleArtefacts(context.Background())


This would basically translate to the OCI list operation, which is often really slow and not scalable.

sounds good, captured in #3076

jonathanj-square added 3 commits October 2, 2024 17:03

- release cli command stubs

63b6c30

- adding docker registry image

- release cli command stubs

e7a480d

- adding docker registry image

- partial oci registry support

02d4279

- integration with CLI to demonstrate release publishing is WIP

ftl-robot mentioned this pull request Oct 4, 2024

Dashboard #728

Open

chore(autofmt): Automated formatting

3ae28f8

stuartwdouglas added the run-all A PR with this label will run the full set of CI jobs in the PR rather than in the merge queue label Oct 4, 2024

stuartwdouglas reviewed Oct 4, 2024

View reviewed changes

docker-compose.yml Outdated Show resolved Hide resolved

jonathanj-square and others added 9 commits October 4, 2024 14:19

publishing artifacts from a memory to registry

66552ff

artefact discovery exposed to the cli

aa8f92c

completed oci Registry implementation

941e5a3

fix cli spec

bc52571

shift external port for registry to avoid conflict

9521ee0

Merge branch 'main' into jonathanj/oci-service-impl

ea77771

chore(autofmt): Automated formatting

2d0618a

collect correct media and artifact type metadata

6b87750

Merge branch 'main' into jonathanj/oci-service-impl

2cc617a

jonathanj-square marked this pull request as ready for review October 9, 2024 20:04

jonathanj-square requested review from a team and alecthomas as code owners October 9, 2024 20:04

jonathanj-square requested review from wesbillman and removed request for a team October 9, 2024 20:04

jonathanj-square changed the title ~~feat: WIP introducing oci artifact registry impl~~ feat: introducing oci artifact registry impl Oct 9, 2024

alecthomas approved these changes Oct 9, 2024

View reviewed changes

stuartwdouglas reviewed Oct 10, 2024

View reviewed changes

This was referenced Oct 10, 2024

Reduce artifact discovery overhead #3076

Open

Spec artifact layout using OCI primitives #3077

Open

jonathanj-square and others added 3 commits October 10, 2024 15:59

applying pr feedback

1d48172

applying pr feedback

f026f30

chore(autofmt): Automated formatting

3d793c4

Merge branch 'main' into jonathanj/oci-service-impl

0a27669

jonathanj-square merged commit b4606de into main Oct 11, 2024
95 checks passed

jonathanj-square deleted the jonathanj/oci-service-impl branch October 11, 2024 18:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: introducing oci artifact registry impl #2989

feat: introducing oci artifact registry impl #2989

jonathanj-square commented Oct 4, 2024 •

edited

Loading

alecthomas Oct 9, 2024

jonathanj-square Oct 10, 2024

alecthomas Oct 9, 2024

stuartwdouglas Oct 10, 2024

jonathanj-square Oct 10, 2024

alecthomas Oct 9, 2024

alecthomas Oct 9, 2024

stuartwdouglas Oct 10, 2024

alecthomas Oct 9, 2024

jonathanj-square Oct 10, 2024

alecthomas Oct 9, 2024

jonathanj-square Oct 10, 2024

stuartwdouglas Oct 10, 2024

stuartwdouglas Oct 10, 2024

stuartwdouglas Oct 10, 2024

jonathanj-square Oct 10, 2024 •

edited

Loading

stuartwdouglas Oct 10, 2024

jonathanj-square Oct 10, 2024

stuartwdouglas Oct 10, 2024

stuartwdouglas Oct 10, 2024

jonathanj-square Oct 10, 2024

stuartwdouglas Oct 10, 2024

jonathanj-square Oct 10, 2024

feat: introducing oci artifact registry impl #2989

feat: introducing oci artifact registry impl #2989

Conversation

jonathanj-square commented Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanj-square Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathanj-square commented Oct 4, 2024 •

edited

Loading

jonathanj-square Oct 10, 2024 •

edited

Loading