[.yeb] add support for easyconfigs in YAML format (ROADMAP)

[boegel]

• using PyYAML: http://pyyaml.org (import yaml)
  • YAML spec with examples: http://www.yaml.org/spec/1.2/spec.html
• implement support in feature/yaml branch

KNOWN ISSUES

• syntax error with URL string values in 'flow context' (e.g. lists)
  • source_urls = [http://www.bzip2.org/1.0.6] doesn't parse
  • see http://pyyaml.org/wiki/YAMLColonInFlowContext
  • workaround: source_urls = ['http://www.bzip2.orig/1.0.6']

ROADMAP

STEP I: parsing of easyconfigs in YAML format

add support for .yeb easyconfigs, step-by-step

• support equivalent of current .eb files

  • bzip2-1.0.6-intel-2015b.eb (minimal easyconfig) [see [.yeb] initial support for parsing easyconfig files in YAML syntax (REVIEW) #1408]
    • initial documentation (see documentation for .yeb format easybuild#161)
  • toolchain easyconfig (dummy toolchain, dependencies) [see [.yeb] easyconfig files in YAML syntax working with dependencies #1429]
  • SQLite-3.8.10.2-intel-2015b.eb (common easyconfig, sanity_check_paths)
  • Python-2.7.10-intel-2015b.eb (dependencies, extensions, local variables)
  • auto-convert (os)dependencies to required type ([yeb] add support for automatic conversion of types in (os-)dependencies  #1442)
  • !majmin YAML function ([WIP] yeb versionmajor function #1444)

• alternate format for toolchain, dependencies?

  • toolchain = intel,2015b
  • dependencies:

  dependencies =
  - [bzip2, 1.0.6]
  - [zlib, 1.2.8], {toolchain: GCC,4.8.2}
  - {name: zlib, version: 1.2.8, toolchain: GCC,4.8.2}
  - {name: foo, version: 1.2.3, versionsuffix: -blah}
  - # external module?
  

• support multiple versions/toolchains/versionsuffixes in a single .yeb file

  • how to express this in YAML? some type of labels/tags/markers?
  • provide .yeb location + ‘specs’ to select (version, toolchain name/version, versionsuffix)\
  • flesh out a couple of examples first, then see how they can be supported

• support collapsing information common to a particular version range, toolchain name, etc.

  • “when building with intel compiler, do this"
  • “for versions > 1.0, do this"

notes:

• issues with typed values: all (non-list/dict) values are strings?
  • True/False, constants (SOURCE_TAR_GZ), ...

STEP II: basic cmdline support for using .yeb files

• translate “eb foo-1.2.3-bar-9.8.7-baz.eb” to “eb foo.yeb” + using version=1.2.3, toolchain=bar,9.8.7, versionsuffix=-baz
• also support alternate ways of specifying what to build
  • “eb foo.yeb(1.2.3,bar-9.8.7,-baz)” (version, toolchain, versionsuffix, use ‘_’ for "don’t care"?)
  • “eb foo-1.2.3-bar-9.8.7-baz” (no .eb file, but “EasyBuild spec”)
  • “eb foo/1.2.3-bar-9.8.7-baz” (module name to install)
  • “eb foo”, “eb [email protected]”, “eb foo{bar}@1.2.3”, "eb [email protected]{[email protected]}-baz” (Spack like)
• make it work with “eb --software=… --toolchain=…"

STEP III: make cmdline feature aware of .yeb: --robot, --dry-run, --try-*

• make --robot consider .yeb files

• make --dry-run aware of .yeb files

  • example:

  * bzip2.yeb # 1.0.6 | _ | intel, 2015b (module: bzip2/1.0.6-intel-2015b)
  

STEP IV: reproducing builds done with .yeb easyconfigs

• copy .yeb + version/toolchain/suffix specs to archive/install dir
  • or rewrite .yeb to include specs (easy to share)?

STEP V: provide script to make the transition from .eb to .yeb

• translate easyconfigs in .eb format to .yeb format
• collapse multiple .eb easyconfigs (for a single software name) into a single .yeb easyconfig

[boegel]

@Caylo is going to spend some time on this during her internship @ HPC-UGent

[fgeorgatos]

extra known issue: conditionals inside easyconfigs won't cut it (ref. CUDA and some others)

[boegel]

@fgeorgatos: all easyconfigs that currently rely on Python code in any way will have to be dealt with differently

[riccardomurri]

Re: conditionals and supporting multiple versions or optional dependencies: .yeb files could be extended using Jinja2 directives, and pre-processed by EB. This is the approach that Salt Stack uses, see e.g. https://docs.saltstack.com/en/latest/topics/tutorials/states_pt3.html

[fgeorgatos]

+1 for jinja2, since it's becoming de facto standard for macro templating
in the python world
On 30 Sep 2015 13:29, "Riccardo Murri" [email protected] wrote:

Re: conditionals and supporting multiple versions or optional
dependencies: .yeb files could be extended using Jinja2 directives, and
pre-processed by EB. This is the approach that Salt Stack uses, see e.g.
https://docs.saltstack.com/en/latest/topics/tutorials/states_pt3.html

—
Reply to this email directly or view it on GitHub
#1407 (comment)
.

[ehiggs]

What is the proposed result? e.g. what does a yeb file look like? e.g. just like the current eb files but with yaml style lists and dicts? Or is something more moving around?

I proposed an idea a while back named ebng which transposed the easy build configuration files in a yaml format to use a delegate system.

!!python/object:easybuild.easyblock.EasyConfig
source: !!python/object:easybuild.easyblock.Retriever.Http
    url: - http://github.com/user/project/tag.zip
         - http://sourceforge.net/no-malware-thx/software-version.zip
configure; !!python/object:easybuild.easyblock.NoOp
make: !!python/object:easybuild.easyblock.Make.CMake
install: !!python/object:easybuild.easyblock.Install.CopyFiles
    files: - cool-exec
           - cool-exec-mpi
           - cool-exec-mpi-openmp
    dest: bin

The advantages here are that we don't need to make a new easy block for each small diversion that each piece of software makes. We just swap it out at the Easy Config level. This can also be built up so common things can swap out the top level EasyConfig. e.g.:

!!python/object:easybuild.easyblock.ConfigureMakeInstall
source: !!python/object:easybuild.easyblock.Retriever.Http
    url: - http://github.com/user/project/tag.zip
         - http://sourceforge.net/no-malware-thx/software-version.zip
configure_flags: '--with-openmp=true'

It also means we can keep loops and some template variables out of the config files since loops in config is hard/impossible to test and hence a massive bug vector.

[boegel]

@ehiggs: for now, the idea is to have support for 'fat' easyconfigs with .yeb in the end, i.e. where we define build specs for multiple software versions and toolchain(s) (versions) in a single .yeb file. We're tackling that one step at a time though, by figuring out how we can support the equivalent of what we have now in .eb in .yeb format, while trying to stick to standard YAML initially (well, more-or-less).

Looking into templating and working towards collapsing easyblocks/easyconfigs like you specify is a way bigger step, but something we can consider in the long run, and we're happy with how .yeb turned out.

[Caylo]

Maybe we can do something like this for dependencies?

dependencies:
    - name: lib, 1.2.8
      toolchain: GCC, 4.8.2
    - name: foo, 1.3
      versionsuffix: -bar

it's a bit longer, but it's clear and more in style with the new toolchain syntax.

[wpoely86]

I like the proposal of @Caylo, readable but not too verbose.

[boegel]

+1, looks very nice, but how does this get parsed exactly?

[rjeschmi]

This is still something that should be reconsidered. I'll add this here as well:
http://jsonnet.org/

[ocaisa]

#2181 is a step in right direction for this, opening the door to dumping yebs to (a set of) easyconfigs at runtime and then using existing resolution methods