(build)dependencies only get loaded after extracting sources

[boegel]

This is a problem when unpacking .iso files using 7z (p7zip), which is generally not available.

Including p7zip as a build dependency doesn't help, as the module only gets loaded after trying to unpack the sources...

So, the prepare_step should be moved up? Or build dependencies should be loaded earlier in a separate step?

[wpoely86]

I would not move the prepare_steps. Seems like a huge can of worms.
We add a prepare_fetch in which we load all the dependencies? So we can use them for both fetching and extracting?

[boegel]

@wpoely86: I'm not sure if that's any less of a can of worms than moving up prepare_step...

Why would moving prepare_step cause problems?

I don't see a real issue in EasyBlock.prepare_step or the easyblocks that customise prepare_step.

The only thing that needs to be handled is making sure that self.guess_start_dir() is done after extracting the sources, so it should move to extract_step.

[wpoely86]

I don't like it. Other people can depend on this order. We have no way to see the repercussions of this. It should only be done in a major version update.

Why would loading the deps before fetch be any trouble?

[boegel]

Well, it wouldn't (it's exactly what's needed), but that would be basically equivalent to moving up prepare_step imho.

The only way that people could rely on prepare_step running after fetch_step is assuming that the sources are already available?

Anyway, if you feel strongly about it, I'm OK with loading the dependencies (i.e., calling self.toolchain.prepare()) before fetching sources without moving prepare_step.

But, I wouldn't add a separate full step for it.

How about defining a set_up_build_environment method (which just calls toolchain.prepare()), and adding it to the list of ready_substeps in get_steps?

[boegel]

I started looking into actually moving prepare_step before extract_step, and quickly had to conclude that @wpoely86 was right: it's too painful; see #3004 .

So, how about this as an alternative: we add support for load_build_deps_before_extract_step = True (disabled by default), which triggers EasyBuild to do exactly that: load (only) the build dependencies (so not including the toolchain) early on, before unpacking the sources.

Since this can be done in a backward-compatible way (by having it disabled by default), there is no need to do this in a major EasyBuild release, and it doesn't impact installations where load_build_deps_before_extract_step is unused (it'll be opt-in, so only used where actually needed).

@wpoely86 Thoughts?

[Micket]

+1 tcsh/csh is another one of those builddeps used as osdeps right now which we could move into a builddep.

[boegel]

This actually goes beyond the extract step.

Sometimes a particular build dependency can be required to download/obtain the source files (like a sufficiently recent git command when git_config is used in the easyconfig file)

[boegel]

There are some creative workarounds possible here, like listing the necessary tools via osdependencies (and install them in the OS), or loading a module in a custom extract command that's specified for a particular source.

This doesn't seem to be important enough to block EasyBuild 5.0 over, and we're not aware of any actual use cases for this.