NF: Add Environment class, with initial Native/Docker implementations

[effigies]

No description provided.

[codecov]

Codecov Report

Attention: 95 lines in your changes are missing coverage. Please review.

Comparison is base (e425b83) 83.42% compared to head (f3c1d35) 81.53%.
Report is 62 commits behind head on master.

❗ Current head f3c1d35 differs from pull request most recent head 0b0c71b. Consider uploading reports for the commit 0b0c71b to get more accurate results

Files	Patch %	Lines
pydra/engine/workers.py	25.35%	53 Missing ⚠️
pydra/engine/environments.py	70.76%	19 Missing ⚠️
pydra/engine/run_pickled.py	23.80%	16 Missing ⚠️
pydra/conftest.py	60.00%	6 Missing ⚠️
pydra/engine/task.py	97.29%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #516      +/-   ##
==========================================
- Coverage   83.42%   81.53%   -1.90%     
==========================================
  Files          22       24       +2     
  Lines        4894     4949      +55     
  Branches     1410        0    -1410     
==========================================
- Hits         4083     4035      -48     
- Misses        807      914     +107     
+ Partials        4        0       -4     

Flag	Coverage Δ
unittests	81.53% <57.58%> (-1.90%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[djarecka]

for now test_docker.py and test_singularity.py might not work, examples for docker env are in test_environments.py
(still more needed)

[ghisvail]

I was able to test this PR successfully on pydra-deface using FSL tasks and a container image built from the latest version of FSL.

A few paths of improvement:

• Although docker is the most "popular", podman is also a suitable alternative. It would be nice to offer the possibility to use any of those somehow. I was able to test podman by directly editing the source code in my test venv.

• On the same vain, the Docker registry is certainly the most popular because it is the default, but it would also be worth being able to specify an alternative registry such as Quay or GHCR for instance.

More info can be found in the containers transport documentation.

[effigies]

I agree, we should definitely try to keep this general. I don't have enough experience with non-Docker/non-Singularity runtimes to be opinionated here, so happy to follow your lead.

[ghisvail]

I did a bit of research:

• Regarding supporting podman in addition to docker, since the former provides a compatible CLI with the latter, Linux distributions either advise using an alias or provide a compatibility package. So it sounds fine to assume the runtime will have a docker command available at first, before thinking of a means to override it if need be.

• Regarding support for other registries, I think the most comfortable is to leave it as part of the image name, i.e. ghcr.io/$username/$image, instead of introducing a registry parameter. Let docker or podman deal with the default registry
  based on their respective settings.

To conclude, I think the current implementation is good enough as it is 👍

[effigies]

@djarecka This is passing locally, but there are a lot of skipped tests. We'll see how far this gets on CI, but this is it from me for today.

[djarecka]

@effigies - ok, thank you. I will review the tests that are failing

[effigies]

Looked at this with some fresh eyes and left a few notes. We should be careful to name the functions and attributes to make future contributors (such as ourselves) less confused.

[effigies]

This is presumably failing in test_docker_inputspec_3. I think we need a clearer use-case and figure out the correct behavior. In any case, we've been putting up with a failing test for a while. I'm inclined to say let's re-add this feature when someone actually needs it.

[djarecka]

i'm trying to understand why the slurm test fails now, since you didn't any new test here yet. any idea?

[effigies]

Nope. I've never understood anything about the slurm tests. I'll rebase on master and push, if that might help.

[ghisvail]

I think the value proposition from the proposed Environment abstraction is clear, I am just a bit confused about the execution.

It sounds to me like they could be simplified to a bunch of context managers, maybe:

# somewhere in the submitter's code
match environment:  # assuming [type, *args]
    case ["docker", image, tag]:
        with setup_docker(task, image, tag) as env:
            ...
    case _:  # fallback to native
        with setup_native(task) as env:
            ...

Here I am using the new pattern matching syntax, but a stack of isintance calls would work the same.

The other worry I have got is how composable the Environment family of values is with the Submitter one. All submitters (cf, Dask, SLURM, ...) should work with native, but would that be the case for containers (Docker, Singularity) or others we had in mind like Conda?

I am totally onboard with the idea to dissociate the execution environment from the task abstraction. But I am still feeling the boundary between Environment and Submitter is not as clear cut than anticipated? I don't have further ideas to improve the proposal at this point, so I won't be opposed to a merge 👍

[effigies]

This is pretty much ready to go. Small comments.

[djarecka]

@ghisvail @tclose - want to try this before merging?

[tclose]

@ghisvail @tclose - want to try this before merging?

Don't have the time atm, so happy to go with this and find problems later