enable --module-extensions by default (+ resolve template values used in extension version)

[Micket]

fixes #4396

[Micket]

I'm very confused, out of the type of tests i expected to see failing, the one that actually broke is test_exts_list, the one the test is supposed to already be using this feature to see the list of extensions?? And It fails because it is missing homepage field.

ERROR: test_exts_list (test.framework.easyconfig.EasyConfigTest)
  File "/tmp/runner/d4b8e7c4de51b3c7733badf96aac3b5c5e83313c/lib/python3.6/site-packages/test/framework/easyconfig.py", line 523, in test_exts_list
    modfile = os.path.join(eb.make_module_step(), 'PI', '3.14' + eb.module_generator.MODULE_FILE_EXTENSION)
  File "/tmp/runner/d4b8e7c4de51b3c7733badf96aac3b5c5e83313c/lib/python3.6/site-packages/easybuild/framework/easyblock.py", line 3778, in make_module_step
    txt += self.make_module_description()
  File "/tmp/runner/d4b8e7c4de51b3c7733badf96aac3b5c5e83313c/lib/python3.6/site-packages/easybuild/framework/easyblock.py", line 1376, in make_module_description
    return self.module_generator.get_description()
  File "/tmp/runner/d4b8e7c4de51b3c7733badf96aac3b5c5e83313c/lib/python3.6/site-packages/easybuild/tools/module_generator.py", line 1302, in get_description
    'homepage': self.app.cfg['homepage'],

I can only assume this test was never actually running properly then? Confusing.

[Micket]

oh, it's just the test that is super misleading, issue has nothing to do with homepage. It was just due to the partial template resolving code that get_description attempts for some reason.

I think it started from needlessly using template to put in the whatis section, and then that template added name, version, homepage and a bunch of other common things since they happened to be there. But this code shouldn't be doing this, it should leave templates alone. It certainly isn't necesssary to add the whatis section, and this change to just write the lines in order directly is shorter, simpler.

[Micket]

I'm struggling with the test suite, because of the already existing bug that breaks --module-extensions=True.

1. The test test_exts_list is implying that this

   easybuild-framework/test/framework/easyconfig.py

   Line 492 in 65591dd

   ("ext-%(namelower)s", "%(version_major)s.0"),

   should be valid.
2. The get_description (and some other parts of module_generator) randomly decided to resolve 3 template strings (name, version, installdir)

   easybuild-framework/easybuild/tools/module_generator.py

   Lines 854 to 857 in 65591dd

   	'name': self.app.name,
   	'version': self.app.version,
   	'whatis_lines': '\n'.join(whatis_lines),
   	'installdir': self.app.installdir,

   and nothing else.
3. If one actually runs the suggested easyconfig, using version_major, it does not get expanded by make_extension_string

   easybuild-framework/easybuild/tools/module_generator.py

   Line 625 in 65591dd

   exts_str = self.app.make_extension_string(name_version_sep='/', ext_sep=',')

   so, the unresolved version_major is there and EasyBuild will crash. Note: This is already present in easybuild, regardless of this PR.
4. Given that %(name) and %(version) is supposed to be resolved to refer to the extension itself, it's a bit confusing to also use it as the name and version for an extension in my opinion. I understand the difference of if it's used for 2 first tuple fields of an extension, I still think it's confusing.

[Micket]

The bug here also is present in the generated EBEXTSLISTxxxx variables inside lua files, where you can find unresolved templates like %(name)s and %(version)s inside the string.

[boegel]

Concrete example of the bug in module for Transformers-4.39.3-gfbf-2023a.eb:

$ grep namelower /modules/all/Transformers/4.30.2-foss-2022b.lua
%(namelower)s-4.30.2, huggingface-hub-0.15.1, regex-2023.6.3,
whatis([==[Extensions: %(namelower)s-4.30.2, huggingface-hub-0.15.1, regex-2023.6.3, safetensors-0.3.1]==])
setenv("EBEXTSLISTTRANSFORMERS", "safetensors-0.3.1,regex-2023.6.3,huggingface-hub-0.15.1,%(namelower)s-4.30.2")

because Transformers itself is included as extension via:

exts_list = [
    ...
    ('%(namelower)s', version, {
        'checksums': ['2586e5ff4150f122716fc40f5530e92871befc051848fbe82600969c535b762d'],
    }),
]

[Micket]

@boegel Actually, looking at this now, i see this was partly fixed in:
#4392
so that module build predates this fix. However, it only does it for the first argument, not the version string, which is what is triggering this bug. So the fix is trivial.

[boegel]

lgtm

[boegel]

lgtm