Overview of technical work planned for the GESIS collaboration

[consideRatio]

Technical work to be done

This is a technical summary of what me and Yuvi believed were good initial steps to take, and are outlined to track work progress.

• Help jupyterhub/binderhub to get semver2 versioned releases.
  • I raised this topic in the JupyterHub team meeting in January 2023, and there was agreement that we should go for it. Practially, we will make releases of both BinderHub the Python package and the Helm chart tied to each other. So, if one has a breaking change, but not the other, we will still bump both.
  • This will need an initial effort, but should be easier once 1.0.0 is released, which is the next release planned.
  • Semtantic versioning of both BinderHub as a package and the Helm chart jupyterhub/binderhub#1408
• Adjust 2i2c basehub to include a new set of templates to represent the binderhub service chart, as extracted from binderhub's chart. This is then later meant to be extracted.
• Deploy a z2jh hub + binderhub service chart* next to it
• See if we can upstream a way to opt-out of making having the /build endpoint provided by binderhub automatically also try to launch a server.
• Explore and figure out what coupling is relevant for what circumstances to couple the binderhub software to the jupyterhub. Note that we believe no coupling is required to just let users get an image built without automatically launching the image.

Terminology

*When I write about a binderhub service chart, I mean a new chart that is only meant to deploy a binderhub deployment (1 pod) and image-builder pods (1 pod per node I think) that the binderhub pod can delegate work to in order to build images.

[jmunroe]

@jmunroe and @consideRatio had chat about current work on this project.

A question that needed an answer was "how tightly coupled should this new persistent binderhub deployment (as described above) be to the existing 2i2c deployment infrastructure"?

If we to deploy it within the exisiting 2i2c infrastructure that would mean

1. Making modifications to basehub (and potential complexity arising from that)
2. Any results obtained would be more difficult to apply to a non-2i2c infrastructure.

Since we want the GESIS collaboration to lead to something that benefits the Jupyter community broadly, we agreed that this project needs to implemented as independent from the existing 2i2c infrastructure.

The next immediate steps are

• Create up a new "seed GitHub repository/test bench" to host the helm charts for this new work (starting with a vanilla z2jh configuration)
  • GitHub repo: https://github.com/2i2c-org/binderhub-service
• Deploy this test bench within a cloud account owned by 2i2c but on a separate kubernetes cluster from our existing 2i2c infrastructure
  • GCP Project: binderhub-service-development
• Add to this test bench the BinderHub software (but not the BinderHub charts) in such away that that binderhub does not automatically start
• To encourage the development of this test bench as a free standing open-source project, create intial scaffolding for both documentation and a testing framework.

We'll have a sprint planning meeting tomorrow within 2i2c and confirm this is a reasonable work to complete within the next two-week cycle.

(@consideRatio Please correct me if I am misrepresenting what we discussed)

[consideRatio]

Hi @jmunroe and @arnim! I wanted to share a status update!

I've worked a lot in https://github.com/2i2c-org/binderhub-service to bootstrap it with all the things I expect from a well maintained project from the start. Here are some things worked and some related things that remains to work.

• Define an initial Helm chart's templates to refine to a functional chart
• Define an initial Dockerfile to run binderhub the software to refine to a functional image
• Automation and documentation for making a proper release, and automation for development releases. For a Helm chart project, this includes:
  • Configure tbump to update version fields etc
  • Document release process in RELEASE.md
  • Building docker images automatically and pushing them with pre-configured credentials.
    • Setup a image repository at quay.io/2i2c/binderhub-service, and provide credentials to push to it
    • Verify established automation to build tag and push
  • Packaging the Helm chart and pushing it to a Helm chart repository (static website, based on github pages)
    • Initial setup with the tool chartpress
    • Verify function
• Define GitHub workflows to automate updates of a frozen requirements.txt environment based on an unfrozen requirements.in file for the Dockerfile
• Setup Dependabot to update misc github workflow's automation
• Setup pre-commit to autoformat markdown, yaml, and python files
• Setup a values.schema.yaml file and automation to build a values.schema.json file for the Helm chart
• Setup a system to handle deprecation of helm chart configuration (In binderhub-service/templates/NOTES.txt)
• Setup documentation system
  • Setup a RTD project to host built documentation
  • Setup automated tests of links and provide pull request previews
• Setup test system
  • Lint and validate chart templates, including tests against the schema
  • Test startup in k8s
• Define an open source license for the project

I've got quite far in initializing a seed for an open source project centered around a Helm chart, some more work remains. After the checkboxes above are checked, I'll start tracking work by making github issues in https://github.com/2i2c-org/binderhub-service and start working to resolve them.

[arnim]

Thank you @consideRatio

binderhub-service is indeed a nice seed template for a project. I think what we need is time to discuss in which direction we want the seed to grow. I really would love to hear your thoughts and I think it's wroth not just to brainstorm one direction (at least in the beginning). @jmunroe and I have also talked about this yesterday :)

[arnim]

BTW @consideRatio would jupyterhub/binderhub/pull/1408 be blocking if we would like to preserve the option to have a shared image repository with a federated BHub?

[arnim]

For reference, there is a conversation with @jmunroe, @consideRatio and @MridulS in the specific binderhub-jupyterhub 2i2c Slack channel on “Dynamic image building in a JupyterHub” and the “persistent binderhub”.

https://2i2c.slack.com/archives/C03RLNFM43F/p1675082222259319

[jmunroe]

@arnim @consideRatio @MridulS Let's try and use part of the time at next week's JupyterHub/Binder collab cafe to connect on this project:

• First Collab Cafe - March 2023 jupyterhub/team-compass#633

[arnim]

Thank you @jmunroe :)

Is there anything I can do?
@consideRatio do you have time to prepare this? Should / can we meet before to discuss what we want to achieve?

[arnim]

In the long(er) run I would also love if @rgaiacs could join :)
However, for the time being he is working on stabilising our deployment to productively rejoin the fed.

[GeorgianaElena]

Hey everyone! Excited to be joining your efforts into this project during the next months 🎉

@consideRatio, I will self-assign to this issue, so I can keep track of it. I plan to start onboarding myself into the project starting Monday, and track down all the async resources available, but I would really love a sync chat with you sometime next week to get more clarity and understand better the goals/assumptions and current state of things.
Let's sync on Slack next week about this 🚀

[damianavila]

Closing in favor of continuing the conversation in 2i2c-org/binderhub-service#27 (to reduce duplication of information and confusion).