Debrief OpenHackWeek

[yuvipanda]

OceanHackWeek has finished its event phase (I think) and the hub (deployed via #549) should no longer see heavy usage. We should debrief after the event, and try to answer the following questions (based in part from #2600):

# Questions to answer

## What went well

## What went poorly

## Speed bumps

## Where we got lucky

## Action items

### Process improvements

### Documentation improvements

### Technical improvements

Tasks

• Wait for the OHW team to have their debrief
• Get some feedback from them
• Decide if we want to discuss with them
• Feedback here: Debrief OpenHackWeek #595 (comment)

[yuvipanda]

@ocefpaf @abkfenris Can you tell us how the hub was for the hackweek, and how the whole process of getting it set up was? Would love to use it to improve our process

[ocefpaf]

Do you want to try to setup a call or should we just comment here? We'll have a OHW debrief soon do you mind waiting for that so we can relay the voices of all the instructors and organizers?

[abkfenris]

Some initial thoughts before we have a full team debrif:

During the week itself, it ran great. I think the only complaints were of the 'why are we doing this on the hub' variety (more from an unfamiliar environment), than anything to do with performance, and those are to be expected.

From my view, I think the biggest issue was the early communication as we transitions from email conversations about cost and feasibility to the technical orchestration in #549 . I know there were some hiccups on the email side from various vacations that were happening on both sides. It appears that some orchestration had already happened behind the scenes based on #554 . Even if it was before our listed start date, it would have been nice to know that things were planned or already happening. The listed start date was also for when we expected to give users access to the hub, which would have required admin access sooner.

It would have been nice to have some documentation for Dask. Since access to Dask resources seems to vary so much, it felt like I had to reverse engineer 2i2c's default image to figure out how to connect. Similarly knowing if there are other requirements that we need to setup our image, and what the (current) constraints of the environment are #549 (comment)

If there had also been some earlier technical scoping conversation, we could have pointed to our old z2jh repo, as I believe we had the same attempts to figure out the right amount of memory last year.

We really didn't need it, but it would be useful for events to know who is usually available when. It wouldn't make sense for us to panic message if everyone is asleep.

With the worldwide nature of the event, we also had 3 of us who were the core of our infrastructure team, so it would probably be helpful to recognize all of us as community representatives, and be included in the templated admins. Also @paigem do you have any comments from the other side of the world?

Getting user data into shared storage took some creativity. It looks like there is potentially a bucket that we could have used, but there wasn't any documentation that I could find past some comments in issues. Instead I largely depended on users sending data to me via Google Drive, Dropbox, or some weird service that I've got a sneaking suspicion in run by Australia's version of the NSA... and then I uploaded that to the shared-read-write directory for them. If there was a data inbox, or a way that we could access their user directories and the shared-read-write dir at the same time, it would allow them to upload data, and we could make the determination of what went out to the rest of the hub.

It would be really nice if we could push images to Google Artifact Registry in order to speed up pulls, and reduce the need for the prePuller and having additional nodes scaled up (which can now be scaled down, and people can wait longer). I did hear about one or two people who had a timeout when trying to start up a server, but they were able to retry and get back in.

[abkfenris]

Ooh, it would also be awesome if we could could give access to the oceanhackweek org or specific teams, which could probably go into the template seeing as jupyterhub/oauthenticator#449 just landed.

[abkfenris]

Also, it would be nice if we could automate updating the image. There may have been a time or two that we merged a PR for our image, then promptly forgot about it while the image built, only to find the build logs hours later.

[paigem]

Great feedback @abkfenris! I don't have anything specific to add, as I hardly did any admin myself since @abkfenris and @ocefpaf did such a great job staying on top of everything. I'll highlight @abkfenris's comments about making the language clearer about the start date of the hub and more info about sharing user data.

But overall, the hub ran really smoothly and was a great shared platform for us all to use!

[abkfenris]

The OceanHackWeek team is meeting on the 18th, and we will hopefully be able to debrief the hub then. I've added this to our discussion topics.

[choldgraf]

@abkfenris thanks very much for this update! It would be really helpful to understand anything that could be improved, either for the hub infrastructure itself, the hub setup and coordination process, the teaching process (e.g., what could have made things easier for users or instructors), etc. We'll appreciate any feedback you're willing to give :-)

I'm also adding a little task list to the top of this issue, so it's clear that our next step is to hear back from the OHW team 👍

[choldgraf]

hey @abkfenris - any updates from the meeting? :-)

[abkfenris]

We ran out of time in our meeting to debrief the hub after trying to get through discussions about what worked & what didn't for a hybrid event, how did the projects and tutorials work out, and when are we doing it again, how to fund it, and what do we want it to look like?

So instead I've got folks sending me feedback, but so far it sounds like we've got most of what people are thinking of was already mentioned, but in general folks were stoked on how well it worked during the event.

[ocefpaf]

Probably worth mentioning that at least one student is using the hub for her PhD work due to lack of resources in her institution. This was helpful beyond our event.

[choldgraf]

I'm gonna close this one as we had a nice writeup of thoughts here: #595 (comment)