Read env vars in extends"

[zupo]

Given the following buildout.cfg

[buildout]
extends = buildout.d/${LEVEL}.cfg

I would like to be able to do

export LEVEL=staging
bin/buildout

with the result being that buildout would run with buildout.d/staging.cfg config. However I get the following error

$ bin/buildout 
While:
  Initializing.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/foo/.buildout/eggs/zc.buildout-2.5.0-py2.7.egg/zc/buildout/buildout.py", line 1991, in main
    user_defaults, command, args)
  File "/home/foo/.buildout/eggs/zc.buildout-2.5.0-py2.7.egg/zc/buildout/buildout.py", line 231, in __init__
    data['buildout'].copy(), override, set()))
  File "/home/foo/.buildout/eggs/zc.buildout-2.5.0-py2.7.egg/zc/buildout/buildout.py", line 1626, in _open
    downloaded)
  File "/home/foo/.buildout/eggs/zc.buildout-2.5.0-py2.7.egg/zc/buildout/buildout.py", line 1587, in _open
    fp = open(filename)
IOError: [Errno 2] No such file or directory: '/home/foo/bar/buildout.d/${LEVEL}.cfg'

[mauritsvanrees]

Maybe https://pypi.python.org/pypi/gocept.recipe.env helps here? It automatically reads the environment variables so you can access them from buildout parts. I use this in a couple of projects. It does not seem to be working when you try to access them from the main [buildout] part though. It seems logical that first the [buildout] part needs to be determined by zc.buildout and then the other parts.

Technically, I guess you could take gocept.recipe.env as an example and turn it into a buildout extension instead of a recipe.

In your example you extend LEVEL and the error says FOO. That is a typo and they are meant to be the same, right?

Easier is probably to not try to let buildout.cfg extend staging.cfg but the other way around. But you might have reasons why that is no option in your case.

[zupo]

Hey Maurits! Thanks for the tips!

In your example you extend LEVEL and the error says FOO. That is a typo and they are meant to be the same, right?

Typo, fixed.

Easier is probably to not try to let buildout.cfg extend staging.cfg but the other way around. But you might have reasons why that is no option in your case.

Yeah, my use-case is much more convoluted, this is just an example to showcase the functionality :)

[do3cc]

@zupo Guessing your goal, are you aware that nowadays you can use env variables in zope.conf?
So if, for example, you want to use env vars to define that the zeo client connects to either prod or dev, you can do that in the recipe and the configuration happens during start up of zope?

[leorochael]

@zupo a major issue with your example is that buildout doesn't seem to take expansion variables into account when working with the buildout:extends key.

I suppose it would be too confusing considering that an extended configuration, or your own ~/.buildout/default.cfg could (re)define variables that change which files would actually be used for extension.

For keys other than extends and extends-cache, it is possible to use environment variables (indirectly) in the [buildout] section (and other sections as well), by passing these variables through the command line.

For example when invoking with the following command line:

export PART=py
bin/buildout buildout:PART=$PART

This configuration works perfectly:

[buildout]
parts =
    ${:PART}

[py]
recipe = zc.recipe.egg
interpreter = py
eggs =
  ipython

[zupo]

@do3cc: nope, not a Zope project I need this on (it's pyramid)
@leorochael: thanks for the tip, I guess this can be my plan B, if all else fails.

[vietdt]

@zupo I've created a buildout extension buildout.extendssubs that provides new option extends-subs which works exactly the same as extends option and also can substitute assignment variables.

[buildout]
extensions = buildout.extendssubs
extends-subs = ${:LEVEL}.cfg

Env vars should be passed via command-line assignment options as in @leorochael's tip.

$ export LEVEL=staging
$ bin/buildout buildout:LEVEL=$LEVEL

Otherwise, I have a patch that update the buildout's _open() function to do substitution for extends option directly. It would be cool if the patch could be merged upstream.

diff --git a/src/zc/buildout/buildout.py b/src/zc/buildout/buildout.py
index e220ee7..37ac8b7 100644
--- a/src/zc/buildout/buildout.py
+++ b/src/zc/buildout/buildout.py
@@ -1624,7 +1624,7 @@ def _open(base, filename, seen, dl_options, override, downloaded):
         dl_options = _update_section(dl_options, result['buildout'])

     if extends:
-        extends = extends.split()
+        extends = _sub_extends(extends, override.copy())
         eresult = _open(base, extends.pop(0), seen, dl_options, override,
                         downloaded)
         for fname in extends:
@@ -1635,6 +1635,17 @@ def _open(base, filename, seen, dl_options, override, downloaded):
     seen.pop()
     return result

+def _sub_extends(extends, override):
+    """Apply substitution for extends option
+    """
+    data = _unannotate(dict([('buildout', override)]))
+    data['buildout']['extends'] = extends
+    data_ops = Options(data, 'buildout', data['buildout'])
+    data['buildout'] = data_ops
+    # utilize Options' _sub
+    ops = Options(data, 'buildout', data['buildout'])
+    return ops.get('extends').split()
+

 ignore_directories = '.svn', 'CVS', '__pycache__'
 _dir_hashes = {}

[zupo]

Passing env var as commandline-assignments is not what I am aiming for and does not really solve my use-case. I mean it does provide an alternative, but since I have many such env vars, the commandline command would get too long to be readable.

I need the env vars to be read directly, from the inside the buildout.cfg file.

[vietdt]

@zupo I think using buildout.extendssubs together with an additional recipe such as collective.recipe.environment or gocept.recipe.env will make things work as expected.

[buildout]
extensions = buildout.extendssubs
extends-subs = ${env:LEVEL}.cfg

[env]
recipe = collective.recipe.environment

export LEVEL=staging
bin/buildout

[leorochael]

@zupo, without seeing why you need to try to select between buildouts with your environment variable, it's difficult to offer you alternatives.

As others have mentioned, you can use environment variables inside any buildout value, with the help of collective.recipe.environment or gocept.recipe.env.

So your "I have too many variables" issue can be dealt with by using those recipes, except for selecting a different file for the buildout:extends option.

However, you could use env. variables to select the a buildout configuration to run by calling:

$ bin/buildout -c ${LEVEL}.cfg

And from all files that could be selected by ${LEVEL}.cfg you start with

[buildout]
extends = buildout.cfg

If you don't want to keep remembering to use the -c switch every time you run buildout, then you can:

• rename buildout.cfg to base.cfg in your repository and your ${LEVEL}.cfg files

• put /buildout.cfg in .gitignore

• and use a command like this in your build script / instructions before running buildout:

  $ cp ${LEVEL}.cfg buildout.cfg

So there are plenty of existing ways of achieving what you need. You just need to use the tools already at your disposal.

[zupo]

We have gone too specific with this issue. The "level" example I provided is just one of many use cases where using env vars in extends is handy.

The other example is loading the configuration from a URL protected by HTTP Basic Auth, but you don't want credentials committed to repository. Something like this:

[buildout]
extends = https://${USER}:${PASS}@my.internal.code.repository/my-project/base.cfg

...

[zupo]

@leorochael I am aware of the workarounds you describe and am using them. But they are workarounds. I like to have my stack as clean as possible. And having the ability to use env vars in extends could potentially remove some of indirection and complexity from my builds.

[zupo]

I'm not using buildout anymore, so I'm closing this.

[mauritsvanrees]

@zupo Out of curiosity, what are you using these days?

[zupo]

@mauritsvanrees: pipenv

It's nowhere near the feature-set of buildout, but I no longer need most of the features buildout provides (due to hosting on Heroku). And for people outside of the Plone community, pipenv is much easier to pick up.

[mauritsvanrees]

I already thought you might have gone for pipenv yes. Thanks for sharing.