Make builddependencies iterable. #2741

bartoldeman · 2019-01-30T20:52:24Z

This is a list of list of buildependencies. It gets iterated on top
of the common builddependencies. This will allow e.g. to build with
multiple Python versions. A toy test is included.

The "prepare" step is now fully iterated which needed a few adjustments.
Hopefully this doesn't break anything.

This is a list of list of buildependencies. It gets iterated on top of the common builddependencies. This will allow e.g. to build with multiple Python versions. A toy test is included. The "prepare" step is now fully iterated which needed a few adjustments. Hopefully this doesn't break anything.

houndci-bot

Some files could not be reviewed due to errors:

./easybuild/framework/easyconfig/easyconfig.py:201:-470: W605 invalid escape ...

./easybuild/framework/easyconfig/easyconfig.py:201:-470: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-461: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-346: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-269: W605 invalid escape sequence '\ '

houndci-bot

Some files could not be reviewed due to errors:

./easybuild/framework/easyconfig/easyconfig.py:201:-470: W605 invalid escape ...

./easybuild/framework/easyconfig/easyconfig.py:201:-470: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-461: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-346: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-269: W605 invalid escape sequence '\ '

houndci-bot

Some files could not be reviewed due to errors:

./easybuild/framework/easyconfig/easyconfig.py:201:-470: W605 invalid escape ...

./easybuild/framework/easyconfig/easyconfig.py:201:-470: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-461: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-346: W605 invalid escape sequence '\ '
./easybuild/framework/easyconfig/easyconfig.py:201:-269: W605 invalid escape sequence '\ '

iterate_builddependencies is a list of lists of tuples unlike exts_list which is just a list of lists.

…ere.

This way we make sure they are always considered.

easybuild/framework/easyconfig/tweak.py

bartoldeman · 2019-01-31T13:14:24Z

This works but still needs a bit of cleanup:

explore keeping iterate_builddependencies separate at all times, and just iterate over it instead of merging it into the common builddependencies.
the resetting of the environment from ready_substeps is tricky. Ideally you'd just unload the module and its deps from the iterated builddependency but that can be tricky. It does state "creating build dir, resetting environment" but I don't quite understand how env.reset_changes() works. @boegel if you can enlighten me when you have time, please let me know.

bartoldeman · 2019-01-31T16:30:23Z

At the EUM @boegel was asking about why not to extend builddependencies. This is tricky because if you do e.g.

builddependencies =
[[('Python','2.7.15'), ('CMake', '3.12.3')],
  ('Python','3.6.6'), ('CMake', '3.12.3')]]

How do you deal with the common CMake builddependency? Parse it twice? Implement duplicate detection in the framework?

…cies.

easybuild/framework/easyconfig/easyconfig.py

…onment.

bartoldeman · 2019-01-31T18:06:15Z

I have now implemented proper module unloading. Hopefully it works fine with Tmod.

Actually the way things are now it may not be to hard after all to iterate over builddependencies itself.
There's some duplication but probably little time is spent vs the actual build.

bartoldeman · 2019-02-01T12:49:05Z

Just in case, I've put a patch vs develop for builddependencies without the iterate_builddependencies here keyword here:
https://gist.github.com/d4f03ab150e65237f5cbdf501e93b818

I think it works well but the hiddendependencies code is getting rather cumbersome, so I'll work on a new PR to address that (I suppose we could set dep['hidden']=True instead of needing to remove them and then adding them back in the dump code).

bartoldeman · 2019-02-01T15:42:45Z

#2748 cleans up hiddendependencies.

…builddep

This reverts commit 3bf5873.

easybuild/framework/easyblock.py

boegel · 2019-03-06T20:25:23Z

easybuild/framework/easyconfig/easyconfig.py

+        builddeps = self['builddependencies']
+        if 'builddependencies' in self.iterate_options and not isinstance(builddeps[0], dict):
+            # flatten if not iterating yet
+            builddeps = flatten(builddeps)


@bartoldeman During today's conf call you mentioned that one option could be to create two separate methods for this, to clearly discriminate the two situations, and avoid mixing them?

easybuild/framework/easyconfig/easyconfig.py

This allows (build)dependencies() be flattened strictly outside the iterating process.

boegel · 2019-03-14T21:18:45Z

@bartoldeman Problem with failing tests with Python 2.6 should be fixed with #2807, but seems like you broke something else as well.

boegel · 2019-03-15T09:07:42Z

(close/re-open to trigger new Travis run now that #2807 is merged)

…dencies

boegel

partially fixed with ComputeCanada#15

boegel · 2019-03-15T15:26:05Z

easybuild/framework/easyconfig/easyconfig.py

+            # take into account that this *dependencies parameter may be iterated over
+            if key in self.iterate_options:
+                deps = flatten(deps)
+                deps_ref = flatten(deps_ref)


@bartoldeman This looks wrong to me... flatten will create a new list, so deps_ref is most definitely no longer a reference to the original list...

Actually, it's OK. deps_ref will be a new list, but the entries inside are still references to the original dependency dict values that we aim to modify, so it's all good...

easybuild/framework/easyconfig/easyconfig.py

easybuild/framework/easyblock.py

boegel · 2019-03-15T15:27:07Z

easybuild/framework/easyconfig/tweak.py

+
+        if key in parsed_ec.iterate_options:
+            val = flatten(val)
+            orig_val = flatten(orig_val)


@bartoldeman Same here, flatten creates a new list, so this is no longer a reference to the original list...

Same here, it's OK to use flatten here, since we're still using references to the original dict values, and that's what wer'e changing (not the list itself).

rename 'EasyBlock.reset_changes' to 'EasyBlock.reset_env' + use 'nub'

…endencies

can't use nub to filter out duplicates in flattened list of build dep…

…ld take a (Python) programming class...)

fix filtering of duplicates in builddependencies method

boegel · 2019-03-15T17:50:39Z

Tested with modified Eigen easyconfig, which lists two different versions of CMake (there's no point in that, other than testing the support for iterating of a list of lists of build dependencies implemented here):

name = 'Eigen'
version = '3.3.7'

homepage = 'http://%(namelower)s.tuxfamily.org/index.php?title=Main_Page'

description = """Eigen is a C++ template library for linear algebra: matrices, vectors, numerical solvers,
 and related algorithms."""

# only includes header files, so no need for a non-dummy toolchain
toolchain = {'name': 'dummy', 'version': ''}

source_urls = [BITBUCKET_SOURCE]
sources = ['%(version)s.tar.bz2']
checksums = ['9f13cf90dedbe3e52a19f43000d71fdf72e986beb9a5436dddcd61ff9d77a3ce']

# stick to latest CMake 3.9.x, since more recent CMake versions require a C++ compiler that supports C++11,
# which may not be available yet in older OSs (e.g. CentOS 6.x)
builddependencies = [
    [('CMake', '3.9.6')],
    [('CMake', '3.12.1')],
]

moduleclass = 'math'

Works as intended, so I consider this good to go.

Thanks you very much for your hard work on this @bartoldeman, I'm well aware that this was far from trivial...

boegel · 2019-03-15T21:55:07Z

easybuild/framework/easyconfig/easyconfig.py

+        # create a list of all options that are actually going to be iterated over
+        # builddependencies are always a list, need to look deeper down below
+        self.iterate_options = [opt for opt in ITERATE_OPTIONS
+                                if opt != 'builddependencies' and isinstance(self[opt], (list, tuple))]


@bartoldeman I only realize now that this is done too early...

We're assuming here that once the easyconfig file is parsed, we know whether which easyconfig parameters are to be iterated over. That breaks backwards compatibility in some sense, since before this change we had the ability to tweak the easyconfig parameter values before they were actually being used...

And that's exactly what we're doing with FFTW, where we define the list of configure options to iterate over in self.cfg['configopts'] via run_all_steps...
That value is not picked up for defining self.iterate_options, and hence stuff is broken.

I think we'll need to postpone this to avoid breaking backwards compatibility (yet still handle builddependencies here)...

problem should be fixed with #2810

houndci-bot reviewed Jan 30, 2019

View reviewed changes

bartoldeman added 7 commits January 30, 2019 21:03

Make docstring raw to make Hound-ci happy.

e723d4b

Fix dump of exts_list by special casing iterate_builddependencies.

fc0b990

iterate_builddependencies is a list of lists of tuples unlike exts_list which is just a list of lists.

Modify toy-0.0-iter dep to have both easyconfig and module available.

26259c7

Restore initial environment when resetting the environment.

66ce0c7

In reset environment for iterate reset to the env first encountered h…

8c91557

…ere.

Avoid temporary modification of builddeps for _finalize_dependencies()

8dd69f3

Merge iterate_builddependencies into DEPENDENCY_PARAMETERS.

5c3beb2

This way we make sure they are always considered.

houndci-bot reviewed Jan 31, 2019

View reviewed changes

easybuild/framework/easyconfig/tweak.py Outdated Show resolved Hide resolved

easybuild/framework/easyconfig/tweak.py Outdated Show resolved Hide resolved

bartoldeman changed the title ~~Add iterate_builddependencies easyconfig option.~~ Add iterate_builddependencies easyconfig option. (WIP) Jan 31, 2019

Fix typo in parced.

a22dfb2

Fix check for iterate_* keys in validate_iterate_opts_lists

f976245

boegel added this to the 3.9.0 milestone Jan 31, 2019

boegel added the enhancement label Jan 31, 2019

bartoldeman added 2 commits January 31, 2019 17:22

Keep iterate_builddependencies completely separate from builddependen…

f074e51

…cies.

Use flatten function instead of list comprehension to flatten.

f5def9e

houndci-bot reviewed Jan 31, 2019

View reviewed changes

easybuild/framework/easyconfig/easyconfig.py Outdated Show resolved Hide resolved

houndci-bot reviewed Jan 31, 2019

View reviewed changes

easybuild/framework/easyconfig/easyconfig.py Outdated Show resolved Hide resolved

houndci-bot reviewed Jan 31, 2019

View reviewed changes

easybuild/framework/easyconfig/easyconfig.py Outdated Show resolved Hide resolved

bartoldeman added 2 commits January 31, 2019 17:58

Properly unload modules for iterate instead of messing with the envir…

3bf5873

…onment.

Make Hound happy about comment.

641421b

Merge remote-tracking branch 'github_hpcugent/develop' into iterable-…

d9f06e3

…builddep

Restore environment when iterating instead of use modules.

4ff5722

This reverts commit 3bf5873.

boegel requested changes Mar 6, 2019

View reviewed changes

Introduce "iterating" easyconfig field.

aa29163

This allows (build)dependencies() be flattened strictly outside the iterating process.

bartoldeman added 4 commits March 15, 2019 03:02

Correct boolean check for iterating.

72c48cd

Remove duplicates in flattened lists of builddependencies.

e8b393b

Fix typo in parameters spotted by @boegel.

b0579ad

Add comment about reset_environ.

248dc25

boegel closed this Mar 15, 2019

boegel reopened this Mar 15, 2019

boegel added 2 commits March 15, 2019 16:08

rename 'EasyBlock.reset_changes' to 'EasyBlock.reset_env'

bddaa0d

use 'nub' to filter out duplicates from flattened list of build depen…

7c34a92

…dencies

boegel mentioned this pull request Mar 15, 2019

rename 'EasyBlock.reset_changes' to 'EasyBlock.reset_env' + use 'nub' ComputeCanada/easybuild-framework#15

Merged

boegel requested changes Mar 15, 2019

View reviewed changes

bartoldeman and others added 2 commits March 15, 2019 11:35

Merge pull request #15 from boegel/iterable-builddep

8e4299b

rename 'EasyBlock.reset_changes' to 'EasyBlock.reset_env' + use 'nub'

can't use nub to filter out duplicates in flattened list of build dep…

96c253b

…endencies

boegel mentioned this pull request Mar 15, 2019

can't use nub to filter out duplicates in flattened list of build dep… ComputeCanada/easybuild-framework#16

Merged

bartoldeman and others added 2 commits March 15, 2019 12:39

Merge pull request #16 from boegel/iterable-builddep

e9bcc8f

can't use nub to filter out duplicates in flattened list of build dep…

fix filtering of duplicates in builddependencies method (@boegel shou…

4945275

…ld take a (Python) programming class...)

boegel mentioned this pull request Mar 15, 2019

fix filtering of duplicates in builddependencies method ComputeCanada/easybuild-framework#17

Merged

easybuilders deleted a comment from boegelbot Mar 15, 2019

Merge pull request #17 from boegel/iterable-builddep

33c46df

fix filtering of duplicates in builddependencies method

boegel approved these changes Mar 15, 2019

View reviewed changes

boegel merged commit 9dc68dd into easybuilders:develop Mar 15, 2019

boegel reviewed Mar 15, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make builddependencies iterable. #2741

Make builddependencies iterable. #2741

bartoldeman commented Jan 30, 2019

houndci-bot left a comment

houndci-bot left a comment

houndci-bot left a comment

bartoldeman commented Jan 31, 2019

bartoldeman commented Jan 31, 2019

bartoldeman commented Jan 31, 2019

bartoldeman commented Feb 1, 2019

bartoldeman commented Feb 1, 2019

boegel Mar 6, 2019

boegel commented Mar 14, 2019

boegel commented Mar 15, 2019

boegel left a comment

boegel Mar 15, 2019

boegel Mar 15, 2019

boegel Mar 15, 2019

boegel Mar 15, 2019

boegel commented Mar 15, 2019

boegel Mar 15, 2019

boegel Mar 15, 2019

Make builddependencies iterable. #2741

Make builddependencies iterable. #2741

Conversation

bartoldeman commented Jan 30, 2019

houndci-bot left a comment

Choose a reason for hiding this comment

houndci-bot left a comment

Choose a reason for hiding this comment

houndci-bot left a comment

Choose a reason for hiding this comment

bartoldeman commented Jan 31, 2019

bartoldeman commented Jan 31, 2019

bartoldeman commented Jan 31, 2019

bartoldeman commented Feb 1, 2019

bartoldeman commented Feb 1, 2019

boegel Mar 6, 2019

Choose a reason for hiding this comment

boegel commented Mar 14, 2019

boegel commented Mar 15, 2019

boegel left a comment

Choose a reason for hiding this comment

boegel Mar 15, 2019

Choose a reason for hiding this comment

boegel Mar 15, 2019

Choose a reason for hiding this comment

boegel Mar 15, 2019

Choose a reason for hiding this comment

boegel Mar 15, 2019

Choose a reason for hiding this comment

boegel commented Mar 15, 2019

boegel Mar 15, 2019

Choose a reason for hiding this comment

boegel Mar 15, 2019

Choose a reason for hiding this comment