Support lock generation from 1..* files

[bollwyvl]

Thanks for conda-lock! Starting to using it in anger!

A use case I'm currently working on solving with jinja, but would consider PRing (eventually, free time, yadda yadda): generating locks based on multiple "inherited" environment files. Consider:

# env-base.yml
channels:
  - conda-forge
dependencies:
  - python =3.7
  # a cast of thousands...

# env-cpu.yml
channels:
  - conda-forge
dependencies:
  - pytorch-cpu

# env-gpu.yml
channels:
  - pytorch
  - conda-forge
dependencies:
  - pytorch

then be able to:

conda-lock -f env-cpu.yml -f env-base.yml -p osx-64 -p linux-64 --prefix cpu-
conda-lock -f env-gpu.yml -f env-base.yml -p osx-64 -p linux-64 --prefix gpu-

... and get

cpu-conda-linux-64.lock
cpu-conda-osx-64.lock
gpu-conda-linux-64.lock
gpu-conda-osx-64.lock

of course, it gets freakier with env-specific yml, etc. as well as extending this to other environment sources, e.g. #39, #40, #41...

[mariusvniekerk]

This looks like a good idea I assume that the model would be that later environments would have precedence.

The bit that is a bit less obvious is what happens when you have different sets of channels. From a least-surprise perspective i would suggest that we use the ordered "union" of all the channels in the various lock files.

[bollwyvl]

Yeah, channels are (always) a drag. I'll have to think about some concrete cases to test... further, one might want to remove a channel.

Current (naive) prototype (unpushed)

• start with composite channels []
• for each file
  • for each channel in environment channels (reversed)
    • if channel in composite channels
      • remove channel from composite channels
    • insert channel at beginning of list

Pro:

• sorta makes sense?
• at least simple to explain

Con:

• can't remove envs

Alternate 1

given how tricky they are, and how this would mostly be made of already under-specified environments, perhaps it would not be unreasonable to just demand that each file has either:

• no channels, perhaps signaled with an empty list, in which case it does nothing
• a full list of channels, in which case it just replaces the entire list

And hard fail if the final composite has no channels specified.

So in the original example, base would have channels: [] and cpu / gpu would stay as is. Extending that example, for a particular win-64 build, there might be an extra env that includes msys2... this might require an extra file just for cpu-win64 and gpu-win64, which kind of defeats the purpose... but will probably be "more right" by being "less magic".

Pro:

• fewer surprises
• can remove envs

Con:

• more files

Alternate 2

• add parsed comments (boo)

# baseline.yml
channels:
- conda-forge  # priority: 100
- anaconda     # priority: 200

Then later...

# windows.yml
channels:
- msys2  # priority: 150

# torch.yml
channels:
- pytorch  # priority: 50

Perhaps setting # priority: -1 would remove it from the list.

Pro:

• can remove envs

Con

• more parsed comments in yml 👿

[mariusvniekerk]

one solution for the channels might just be

--override-channels --channel a --channel b

on the cli

[bollwyvl]

Yeah, some naive behavior that is easy to override might be the right balance...

[mariusvniekerk]

mostly want to get the set of input formats mostly done and then we can move on to this

[mariusvniekerk]

of the alternates i kinda like alt: 1 aka the last one that specifies anything overrides the whole list

[JorgeGarciaIrazabal]

I am not sure if this is possible, but another alternative can be adding this configuration inside env.yaml. Maybe something like this...

# env-base.yml
channels:
  - conda-forge
dependencies:
  - python =3.7
  # a cast of thousands...
conda-lock: # use  a better name for this
  -  # env-cpu.yml
    prefix: cpu-
    channels:
        - pytorch
    dependencies:
      - pytorch-cpu
 
  - # env-gpu.yml
    prefix: gpu-
    dependencies:
      - pytorch

This way, it is pretty clear that there is a base environment and the other ones would be the union of them, making it very easy to confirm that no channels of dependencies are duplicated.

This approach seem like the most intuitive way for the final user to me (we have a file to define the environment, let's use it to make the environments even if I need more than 1)

The main con here is that we can only have one base env which might be enough not to consider this, but I just wanted to share how I would like to define my environments :P.

[bollwyvl]

Getting the other formats in does seem like the tricky part... But maybe the way it works can be influenced by this future goal. My motivation for them being separate, valid files is that they: - can be used/validated by other things (e.g schema) - can be used to calculate only the needed lock files to change with e.g. make, doit a new monolithic format is a new problem, and changes all envs at once... anaconda-project uses this approach, supporting an "inherit_from" list per env, but it's lock file leaves something to be desired (also monolithic, has no hashes or channel sources per package)

[mariusvniekerk]

Okay we've landed support for channel overrides in master. So it should be a pretty simple PR to get multiple files on if you want to take a stab at it @bollwyvl.

My suggestion to map to which kind of file it is based on

1. filetype
2. if yaml, presence/ absence of certain key fields to distinguish between say meta.yaml/ environment.yaml

[bollwyvl]

aw man, mamba.yml is a thing? I'll give it a shot, probably be this weekend...

[mariusvniekerk]

i think the main way to get the mutifile stuff to work is to make sniffers a-la libmagic but for text

We're not supporting things like boa's recipe.yml yet since that hasn't been stabilized

[bollwyvl]

My kingdom for config files that include `$schema`...

[bollwyvl]

Whoa, thanks! 💜

…

On Fri, Oct 9, 2020, 11:20 Marius van Niekerk ***@***.***> wrote: Closed #42 <#42> via #57 <#57>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALCRCNQ3FNU4WPTHTISNDSJ4S3TANCNFSM4QU5KVUQ> .