[ENH] - Test for parallel builds

[kcpevey]

Feature description

We need the ability to build environments in parallel. This functionality was added in #392 but there was issues and the work needed to be reverted.

We need to re-implement this feature.

Value and/or benefit

Building all environments in series is not efficient and once there are a number of people on the platform, wait times for builds grows to an unacceptable length of time.

Anything else?

No response

[kcpevey]

@costrouc can you help scope this work? Do we need to make some changes to the previous effort or should we take a new direction?

[costrouc]

@kcpevey parallel builds are supported. I don't think any action needs to be taken. We just need a release of conda-store.

I'm going to close this issue unless additional work needs to be done.

[dharhas]

Have the bugs we found during the pycon tutorials been fixed? Have we tested parallel builds in anyway?

[kcpevey]

Are multiple builds of a single environment also parallel? i.e. I click "Create" for an env to be built, immediately realize I forgot a package, edit the yaml, and click "Create" again.

I've done this quite a few times and the second env immediately says "building" but the time it takes to build seems to indicate its waiting for the first build to finish. This is just a suspicion, since I can't really test or see what's happening as an end user.

[dharhas]

reopening until this discussion is resolved.

[kcpevey]

Ok, I've been testing on this today. I have a env that takes over an hour to build, call it slow and I have an env with just one package that takes one minute to build, call it fast.

Note that when you submit an env to be built, it immediately shows "Building" no matter what.

What I observed:

1. Build fast - took 1 minute
2. Start building slow, then start building fast. Waited 30 min, slow was still building, fast was also showing that it was "still" building. Came back even later, both envs were complete. Total build time for slow and fast was the same (via the admin ui).
3. Retested steps 1 & 2 and got the same result.

I do not think parallel builds are working.

[costrouc]

@kcpevey what are you testing on. The main branch of conda-store does support parallel builds.

I do think the action item here is that we need a test that parallel builds work. I think this feature is also worth mentioning in the docs like you said @dharhas.

[kcpevey]

what are you testing on. The main branch of conda-store does support parallel builds.

That's a good point. I'm testing on the nebari instance we have deployed for jatic. We used to be deploying from main, but I dont think that's true for this deployment.

[nkaretnikov]

Added a test in #567. See the discussion there on potential limitations of this test.

[nkaretnikov]

Also, there was this issue regarding DB connections and celery concurrency: #247. Because of this, in all example configs, we set the concurrency value to 4, which might not be enough in some environments(?).

But I'm not sure what the current state is, I haven't looked into this DB issue at all.

[trallard]

@nkaretnikov what is the status with this?

[nkaretnikov]

@trallard A week+ ago, Chris and I agreed this should be tested by making API requests, so that's the new plan. But I haven't started with this so far. Should I prioritize this over all other things assigned to me? Or is there a particular deadline when you want to see this done?

[trallard]

No need to reprioritise other tasks. I was just checking in.
You can take this up back again when you're done with the other items in your backlog 😊