easy_update for R and Python easyconfig

[fizwit]

easy_update will update the exts_lists for Python, R and R-BioConductor using the native API for each language. Updating the version numbers for every package in ext_lists. easy_update also checks for dependencies recursively and adds any missing modules that are dependencies. If you support R and Python easyconfigs with hundreds of modules this tool will save you many hours of work.

[boegel]

@fizwit Thanks a lot for contributing this!

There are a couple of alternate approaches to this (cfr. #1865 and https://github.com/fgeorgatos/easybuild.experimental/blob/master/users/pneerincx/generateEasyConfig.R), but from what I can tell at a first glance this looks like a better approach.

I haven't found time to deep dive into this, but it would be very interesting to include support for something like this in EasyBuild itself...

Are you up for working with us towards that?

[rjeschmi]

We need to manage the dependencies better since it depends on some things that easybuild doesn't (like requests?)

Also it feels like there should be an output of significant changes other than version changes (packages are sometimes added in my R test)

[boegel]

@rjeschmi we could have requests as an optional dependency, that only kicks in when you try to use this feature (we're doing that for other features too, e.g. GC3Pie, etc.)

[rjeschmi]

Just to add things while I'm trying to get it working.

It seemed to miss the plogr dep for RSQLite (or at least RSQLite complained about it). Seems like a new dep...

[fizwit]

apologies for the long e-mail; Use --verbose and grep for "(new)" packages. Yes, dependencies are an issue. Python and R will pull extra package dependencies into a module which do not get documented in the easyconfig file. easy_update processes the exts_list linearly and checks the dependencies for each package. Package dependencies are compared to the package list above the current package in the ext_lists. If you have maintained exts_lists by hand easy_update will "rewrite" your exts_list based on the dependency tree. Caveat: easy_update is only as good as the information that is in CPAN and Pypi. The default output from easy_update only gives a brief summary. Use --verbose argument after the package name. Verbose will output a lot of information but you will get a list of "new" modules. The following example lists Jinja2 and jsonschema as duplicates. jsonschema is flagged as a duplicate package but is needed as a dependency to an other package and is written into the new exts_list above its current location in the list. When jsonschema is encountered again the package is removed from exts_list and is flagged as "duplicate".

easy_update.py Python-3.6.0-foss-2016b.eb

Package name: Python-3.6.0-foss-2016b duplicate: Jinja2 duplicate: jsonschema Updated Packages: 37 New Packages: 6 $easy_update.py Python-3.6.0-foss-2016b.eb --verbose | grep new output WTForms : 2.1 (new) [273, 61] jsonschema : 2.5.1 (new) [273, 192] testpath : 0.3 (new) [273, 267] entrypoints : 0.2.2 (new) [273, 268] nbconvert : 5.0.0 (new) [273, 269] notebook : 4.3.1 (new) [273, 270]

…

On Tue, Jan 17, 2017 at 2:07 PM, Robert Schmidt ***@***.***> wrote: We need to manage the dependencies better since it depends on some things that easybuild doesn't (like requests?) Also it feels like there should be an output of significant changes other than version changes (packages are sometimes added in my R test) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2061 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE4tyz29b26eOefWiBtYVAsrEBXPXGAEks5rTRF5gaJpZM4LPpMf> .

[rjeschmi]

thanks for the extra info. There are certainly some problems somewhere in dependency resolution as I have to manually fix a few packages, but it does save a lot of time already.

I'll keep poking at it.

[fizwit]

Thanks for the catch; Yes indeed, plogr was not listed as a dependancy for RSQLite. plogr was listed in a structure called "LinkedTo". I have updated easy_update to include this additional dependency list from CRAN. I have also updated the verbose output for R to make it a little easier to read. Note: my R buid has 511 packages, without easy_update it is no longer possible to maintain easyconfig of this size.

         RSQLite : requires DBI
         RSQLite : requires Rcpp
         RSQLite : requires memoise
         RSQLite : requires Rcpp
         RSQLite : requires BH
         RSQLite : requires plogr
         memoise : requires digest
         memoise : 1.0.0    (new) [511, 137]
           plogr : 0.1-1    (new) [511, 138]
         RSQLite : 1.1-2    (update) [511, 139]

[boegel]

@fizwit You didn't push the updated version of easy_update into this pull request?

[fizwit]

I have merged my newest version of easy_update.py. Many small bug fixes have been made.
easy_update.py almost passes pep8
LinkingTo object from CRAN to list of R dependencies.
--check [package name] Single package lookup feature added to easy_update (only works with R at this time)
issue#2 resolved

[boegel]

worth mentioning here: easy_update is also available via https://github.com/fizwit/easy_update

I'd still like to see this integrated in the EasyBuild framework, I feel it would be most useful as an eb command line option as opposed to a stand-alone script, since that would make it more directly accessible, and it would help to ensure that the script doesn't get broken (by including tests for it in the framework test suite).

[boegel]

@fizwit There's still interest to get this integrated into the EasyBuild framework, we briefly discussed this during the last conf call...

I'm guessing you're closing this because you're happy with having this as a stand-alone script for now?