Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up docker.cdot.systems Docker Registry #2801

Closed
humphd opened this issue Feb 2, 2022 · 29 comments
Closed

Set up docker.cdot.systems Docker Registry #2801

humphd opened this issue Feb 2, 2022 · 29 comments
Assignees
Labels
type: discussion Requires conversation type: enhancement New feature or request
Milestone

Comments

@humphd
Copy link
Contributor

humphd commented Feb 2, 2022

Following-up from #1743 (comment), we need a solution for pushing docker images from CI to a registry. Our options seem to be:

  1. use the GitHub Package Registry
  2. host our own Docker registry on staging/production

I don't think there is another (free) option. Docker Hub is not going to work for a Team, and Amazon ECR will cost us. Are there other options I'm missing?

If we go with 1., which is probably the simpler of the two, we need to figure out what would break, and what has to be re-created. That includes:

  • webhooks
  • team members, permissions
  • Vercel integration will break, and I don't think we can get it back again (they don't do Teams for free any more, we were grandfathered in)
  • maybe some deployment stuff?

I have no idea what else would break. If we do this, we'd have to plan it carefully and execute it in such a way that it wouldn't cause major downtime for the project/community.

Thoughts?

@humphd
Copy link
Contributor Author

humphd commented Feb 3, 2022

We're discussing doing this on the Thursday before study week, and then fix things over the break.

@humphd
Copy link
Contributor Author

humphd commented Feb 3, 2022

According to https://docs.github.com/en/repositories/creating-and-managing-repositories/transferring-a-repository, this should mostly work:

People will need to update their upstream remote with git remote set-url origin new_url.

I think people with access will still retain it too, though, we might need to tweak permissions and teams. I'm not sure what's going to happen with Projects, especially those that are attached to the Seneca-CDOT org.

@humphd
Copy link
Contributor Author

humphd commented Feb 4, 2022

I looked to see what we could create. The following team names (e.g., XXXX/telescope) are (currently) available:

  • SenecaOpen
  • OpenSourceCourse
  • LearningOpen
  • PlanetSeneca
  • SenecaCodes
  • code4seneca
  • osd-dps
  • SenecaBlogs

Other ideas?

@Kevan-Y
Copy link
Contributor

Kevan-Y commented Feb 4, 2022

I looked to see what we could create. The following team names (e.g., XXXX/telescope) are (currently) available:

  • SenecaOpen
  • OpenSourceCourse
  • LearningOpen
  • PlanetSeneca
  • SenecaCodes
  • code4seneca
  • osd-dps
  • SenecaBlogs

Other ideas?

I love PlanetSeneca name

@sirinoks
Copy link
Contributor

sirinoks commented Feb 4, 2022

  • SenecaOpen
  • OpenSourceCourse
  • LearningOpen
  • PlanetSeneca
  • SenecaCodes
  • code4seneca
  • osd-dps
  • SenecaBlogs

I don't really like having Seneca in name, OpenSourceCourse sounds so generic to be available.. Voting for that

@DukeManh
Copy link
Contributor

DukeManh commented Feb 4, 2022

I really want to stick with Seneca-CDOT, but I'll go with SenecaOpen if I have to choose.

@DukeManh
Copy link
Contributor

DukeManh commented Feb 7, 2022

@humphd Why doesn't Docker Hub work for us? I think they have a free plan for education/open-source.
image

@humphd
Copy link
Contributor Author

humphd commented Feb 8, 2022

We could try. What I don't love about is the fact that it would be tied to my account vs. a team (blocked on me to do any fixes), and that we get rate limited "200 image pulls per 6 hours." Without good caching, this is going to mean that CI builds, deployments, local dev, etc. will just randomly fail.

I don't know, maybe I'm wrong and we should try it first.

@humphd
Copy link
Contributor Author

humphd commented Feb 8, 2022

We could also try https://www.docker.com/blog/expanded-support-for-open-source-software-projects/

@TDDR
Copy link
Contributor

TDDR commented Feb 8, 2022

We could also try https://www.docker.com/blog/expanded-support-for-open-source-software-projects/

This seems like the easier solution, but is easier better?

Do we meet the other two requirements, or should issues be made to address them?

  • Be public and non-commercial
  • Meet the Open Source Initiative (OSI) definition (shown here), including definitions for free distribution, source code, derived works, integrity of source code, licensing and no tolerance for discrimination
  • Distribute images under OSI approved open source license
  • Produce Docker images used to run applications

@humphd
Copy link
Contributor Author

humphd commented Feb 8, 2022

We use an OSI approved license, https://opensource.org/licenses/BSD-2-Clause.

Our images would be under the same license. I think we are good for all of this.

@humphd
Copy link
Contributor Author

humphd commented Feb 8, 2022

Docker Open Source Community Application
Thank you for completing the Docker Open Source Survey. We will review your application, and get back to you as quickly as we can. We have received a large of applications, so please bear with us, as we go through the review process

OK, I filled out the form. I'll see what they say.

@humphd
Copy link
Contributor Author

humphd commented Feb 9, 2022

Another follow-up. I had a good meeting today with Chris Tyler. He and I agreed that if Docker says 'no', or it doesn't make sense to use Docker Hub, we'll look at hosting our own Docker Registry in CDOT. He has a machine with 1/2 TB SSD that we could use pretty quickly, and a lot more NAS storage that we could expand into later if needed.

Running our own registry is well documented at https://docs.docker.com/registry/deploying/. Essentially, we need to run the a registry Docker container, and configure it for proper storage volumes, access, and security.

We might be able to do this using our existing infrastructure, which would get us the TLS and security for free. If not, we'll have to do Let's Encrypt, DNS, etc ourselves.

@cindyorangis
Copy link
Contributor

Oh thank god, we don't have to move out of Seneca-CDOT, I was having an identity crisis over what name Telescope's org was gonna get.

@humphd
Copy link
Contributor Author

humphd commented Feb 10, 2022

This is interesting, re: docker build:

By default, the build cache is based on results from previous builds on the machine on which you are building. The --cache-from option also allows you to use a build-cache that’s distributed through an image registry refer to the specifying external cache sources section in the docker build command reference.

With our own registry, we could re-use the build cache in CI or on development machines too.

@humphd
Copy link
Contributor Author

humphd commented Feb 17, 2022

I've reached out to Chris to start the process of creating our own Docker Registry.

Those that are interested in being involved should begin by reading https://docs.docker.com/registry/deploying/. Once I talk to him and get more details, I'll file specific issues that we can start fixing.

@humphd humphd changed the title Make Telescope project usable by a Docker Registry Set up docker.cdot.systems Docker Registry Feb 22, 2022
@humphd
Copy link
Contributor Author

humphd commented Feb 22, 2022

I've just finished a meeting regarding this, and here's the update:

  • Chris has to go to campus to move some SSDs around so we can take over one of his servers.
  • He's going to create a VM that we can use to host the Docker Registry, and we'll initially have access to a few hundred Gigs of storage.
  • Later, he's hoping to move this into a rack with faster network and access to a larger set of network storage, but that won't happen this term.
  • He hopes to have this set up done by Friday afternoon
  • We'll run the registry at https://docker.cdot.systems

TODO items for this work:

I'm sure there's more to do, but that's the list I have right now. Let me know if and how you want to be involved.

@TDDR
Copy link
Contributor

TDDR commented Feb 23, 2022

I've just finished a meeting regarding this, and here's the update:

* Chris has to go to campus to move some SSDs around so we can take over one of his servers.

* He's going to create a VM that we can use to host the Docker Registry, and we'll initially have access to a few hundred Gigs of storage.

* Later, he's hoping to move this into a rack with faster network and access to a larger set of network storage, but that won't happen this term.

* He hopes to have this set up done by Friday afternoon

* We'll run the registry at https://docker.cdot.systems

TODO items for this work:

* we need to get SSL certs setup for the `docker.cdot.systems` domain with Let's Encrypt, and have them auto-update.  Ideally we can steal/learn from what @manekenpix has done on Telescope (I don't know all the steps)

* we should probably run this behind nginx, which is documented at https://docs.docker.com/registry/recipes/nginx/#setting-things-up

* setup Basic Auth username/password pairs for any service (e.g., GitHub Actions) or person who needs to `push` to the registry.

* we should see if we can set-up pull through mirroring https://docs.docker.com/registry/recipes/mirror/

I'm sure there's more to do, but that's the list I have right now. Let me know if and how you want to be involved.

I would like to be involved. I don't have many issues assigned right now for anticipation of this, so load me up with a decent workload for over the next couple of weeks if possible.

@DukeManh
Copy link
Contributor

What will be the network bandwidth of the current system?

@Kevan-Y
Copy link
Contributor

Kevan-Y commented Feb 23, 2022

Just wondering, I read a tutorial, about self-hosting docker registry. Is it something similar to this https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-18-04 we gonna do?

@humphd
Copy link
Contributor Author

humphd commented Feb 23, 2022

What will be the network bandwidth of the current system?

Eventually 10 Gigabit between the docker registry and staging/prod, but not until we move it into the server rack later in the term. For now, I don't know the answer, but should be pretty fast.

@humphd
Copy link
Contributor Author

humphd commented Feb 23, 2022

Just wondering, I read a tutorial, about self-hosting docker registry. Is it something similar to this https://www.digitalocean.com/community/tutorials/how-to-set-up-a-private-docker-registry-on-ubuntu-18-04 we gonna do?

Yes, very similar, but on Fedora vs. Ubuntu.

@humphd
Copy link
Contributor Author

humphd commented Feb 24, 2022

@TDDR, @rclee91, @Kevan-Y for the docker registry, here is what I think we need to do:

@TDDR I've put your name on all of this, but we can add more people and should split this up into separate issues. I know @rclee91 and @Kevan-Y want to join in as well, so maybe they can speak up. I'm happy to take any of it you don't want and do reviews.

I would suggest trying to create all this for your local fork without SSL to learn how the proxy works (i.e., try to get it setup so you have to authenticate with GitHub on your local machine)

@humphd
Copy link
Contributor Author

humphd commented Feb 24, 2022

OK, server is setup, docker.cdot.systems created. Thank you @ctyler!

Now the fun begins.

@DukeManh
Copy link
Contributor

This is exciting, I added my name to a task.

@Kevan-Y
Copy link
Contributor

Kevan-Y commented Feb 25, 2022

Assigned myself to some tasks too.

@humphd
Copy link
Contributor Author

humphd commented Feb 27, 2022

I think oauth2-proxy might be hard to use for automated login scenarios (e.g., pushing to the registry from CI). I think we should look at https://github.com/cesanta/docker_auth too, which should let us do it with tokens, see https://github.com/cesanta/docker_auth/blob/main/examples/reference.yml#L122-L163

@humphd
Copy link
Contributor Author

humphd commented Mar 9, 2022

Auth discussion happening in DevelopingSpace/docker.cdot.systems#3

@tpmai22
Copy link
Contributor

tpmai22 commented Mar 11, 2022

@TDDR if there by chance you uploaded the PR by tonight, please also file following up issues for 2.9 and close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: discussion Requires conversation type: enhancement New feature or request
Projects
None yet
Development

No branches or pull requests