Ability for pillar to read other pillar values

[alienzrcoming]

Hi, I have a number of states that could be greatly simplified if pillar could read other pillar values. One use case example is I have a "walt_warning" pillar value which i populate in all managed files. I have several pillar values that declare the entire contents of files which are referenced by jinja templates, and instead of having to populate each instances of this with the contents of the salt_warning value, it'd be much easier to just list it as {{ pillar['salt_warning'] }} in the pillar value itself.

Another example would be a situation where pillar declares values that are dependent on another pillar variable the declares the environment a system is in (eg test, staging, live), which is also declared in pillar.

Thanks in advance!

[basepi]

Seems like a chicken and an egg problem to me -- I don't know how we could do this without multiple passes.

However, @thatch45 thinks it might be possible but is not sure how to do it at this point. I've put this on the Approved for Future Release milestone until we find a good way to implement this.

[dimasmjunior]

#4326 may be related.

[malinoff]

I've been thinking about this for a while.

I should agree with @basepi . Pillar is a minion-specific data which is stored on the master and compiled on a minion. There could be a problem: how do I get a pillar data if that pillar is not associated with a minion?

We can't switch compilation to the master side, obviously. And we can't say salt, please, give access to the chunk of a pillar that is not associated with current minion, because it will lead to the serious security problem.

The only possible solution I see is to make something like Pillar external data.
Right now we have a simple top.sls file which just associates pillars with minions.

dev:
  'test-minion':
    - servers

Probably we can complicate this a bit. Let's say we have servers.sls pillar (with any data) and database/info.sls with such content:

hostname: mysql-db
port: 9998
user: username
password: my_cool_password

Thus, we define external_data (probably wrong name) subsection with such content:

dev:
  'test-minion':
    - servers
    - external_data:
      - database.info.user
      - database.info.password

Salt will make sure that defined path (e.g. database.info, without last .user/.password chunks) exists and it's a common file, then it will look through the file (database/info.sls) in order to find user and password data.
If they were found, salt will transfer not all the database/info.sls file, but a file with only user and password data specified.

As i see this should work without any security issues.

[techdragon]

I just want to throw in my experience with this matter. I have a fairly simple use case. I need to setup a postgres DB, and setup an application that uses this DB, naturally both of these will have overlapping values and I would like to declare these in only one location. Normally I would simply declare the data in an agnostic 'settings' dict in the pillar and have the states manage it and both postgres state & application state reffer to the one value in the pillar.

However, I have just begun to investigate using the salt formulas and these complicate the issue.
The postgresql formula is setup to take care of making the DB, yet i cant use it effectively in concert with another formula for an application (without rewriting the formulas) running on postgresql due to the fact that both formulas are expecting their respective pillar files to have a data attribute.

In my case its the sentry formula and the postgresql formula but its another angle to this 'pillar in the pillar' problem and if the solution doesnt fix this, well. Then theres still a bigger problem :-/

[ruimarinho]

I'm facing a similar issue which would really benefit from this feature. Wouldn't this pave the way to setting pillar data via salt states so that they can retroactively affect other related states too?

[jberkus]

Let me add my comment onto this, and an idea for implementation.

The reason I want this is that I have a fairly complex pillar written in "#!py" for a customer. However, there is a minority of the content of that pillar, currently expressed as a dict declaration, which I would like that customer to edit. This would be vastly less error-prone if I could store that dict as a 2nd pillar in pure YAML. However, I can't because pillars can't refer to other pillars.

My implementation idea is simple: load the pillars in the order given in pillar/top.sls. If a user wants a pillar to refer to another pillar, they need to get the ordering right themselves.

[arnisoph]

I really need this feature since I need to assign pillars to minions depending on the environment variable which is (who guesses it) another pillar.

Storing role/ environment information on minion-side in grains is UNSAFE and should NEVER be done. The salt docs mustn't recommend this either. I also don't have access to external pillar data in my pillar files.

We really need a solution here. I don't see a salty solution for this problem yet.

This is also related to:
#1064
#4244

[arnisoph]

As a workaround I'm creating a YAML file with common data and serializing into a (Jinja) template variable as I do it in my formulas: https://github.com/bechtoldt/httpd-formula/blob/master/httpd/init.sls#L1

[alienzrcoming]

@bechtoldt you might find reclass helpful (http://reclass.pantsfullofunix.net/)

i spent some time this weekend rolling out grains as a way to work around this. next i'll be playing with reclass. i keep reading that it provides a way around this limitation.

[arnisoph]

@alienzrcoming I'm using salt.pillar.foreman for this but don't have access to it's within pillar SLS files.

[uvsmtid]

UPDATE: This is impossible approach at the moment because Jinja template is processed before include (the include is part of output YAML data which Jinja passes through without analysing). Inclusion is done by Salt itself after Jinja (even after YAML parsing).
The rest of the post is just a demonstration of failed attempt.

Preserving pillar include order

I thought it already works by preserving order in which pillar files are loaded.

The idea was simple:

• Use include key in pillar files to declare dependency of content within pilar file on content from another pillar file. It's probably similar to how state's include works (it is how import or include works in Python, C/C++, Java, ... for dependencies).
• In "single pillar file" case, pillar keys are loaded from the top to the bottom of the file (just like definitions in most of the programming languages).

Pillar example

# top.sls
base:
    '*':
        - two

# one.sls
the_first: 111

# two.sls
include:
    - one
the_second: {{ pillar['the_first'] }}

Problem confirmed

Currently two.sls fails to compile because Jinja obviously does not care about include keyword:

2015-04-06 22:58:04,519 [salt.pillar      ][CRITICAL] Rendering SLS 'two' failed, render error:
Jinja variable 'dict object' has no attribute 'the_first'

[0xf10e]

I've first thought ext_pillar_first=True would make this work by injecting them before the parsing of the rest of the pillar-files begins.

Something like this might even be an option: Either accumulate pillar-data (first process elements of the list ext_pillars, then elements of pillar_roots) or provide access to external pillars via a variable ext_pillar during processing of pillar_roots (as ext_pillar_first=True gives them a higher priority than pillar_roots anyway).

[mspinassi]

I've done this using Puppet + hiera before, and was a tremendous win.
For example, I'd like to have a file with passwords (only one file with passwords let me easily encrypt it before pushing to git), and then in any other pillar file reference those values.

[uvsmtid]

There is a (workaround) solution!

Recent updates in this issue pointed at loading variables by Jinja itself (which means ability to data in one Jinja file from another) - @wuxxin, thanks!

For example:

• /srv/pillar/path/to/some_paramters.jinja - this file (or many like this) under pillar directory which contains loadable variables imported by Jinja elsewhere later:

{% load_yaml as vars %}
regular_yaml:
    with_arbitrary:
        structure:
            - a
            - b
            - c
{% endload %}

• /srv/pillar/path/to/some_pillar.sls - this is the pillar written in Jinja which loads variables from Jinja file above:

{% from 'path/to/some_paramters.jinja' import vars %}
variables_loaded_from_another_file: {{ vars }}

It's not exactly accessing pillar values (because real pillar values are result of rendering).
But it solves problems of pillar files parameterization:

• data duplication (defining the same parameters in more than one file)
• unnecessarily distributed data like using grains instead (which are minion-specific)
• avoiding use of variables from Salt configuration as described here
• keep all configuration parameters centralized in pillar

Even if it is a workaround, the approach reduces the urgency of implementation.

[mitar]

I was also hit by this. It seems all pillars are combined together and ordered based on alphabetical order. So I can access those which top-level field are before in alphabetical order, but not later. And this is despite me specifying the order in top.sls. I think order in top.sls should be respected.

[mitar]

I take back. It does not seem to be by alphabetical order.

[bbinet]

FYI, I've created the PillarStack ext_pillar which supports to read other pillar data.
See https://github.com/bbinet/pillarstack for more information.

[morsik]

@uvsmtid: I found another (easier?) solution.

Leave original file as it was:

regular_yaml:
  with_arbitrary:
    structure:
      - a
      - b
      - c

In file you want to put include:

{% import_yaml 'path/to/some_paramters.sls' as vars %}
variables_loaded_from_another_file: {{ vars }}

import_yaml is documented here: http://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.jinja.html#salt.utils.jinja.SerializerExtension

[uvsmtid]

@morsik: Yes, it's less cumbersome!

The only caution for those who employ examples above is that import_yaml expects pure YAML only. In other words, *.sls extension is deceiving (as it implies that imported file can be of any format for any renderer like Jinja while it must be pure YAML):

{% import_yaml 'path/to/some_paramters.sls' as vars %}

The same documentation has examples of import-ing Jinja-level variables and macros from one Jinja-template into another. Note the difference of lower parser-level (import_yaml or import_json) and higher template-level (import variables and macros from Jinja code) importing.

Bottom line

Jinja + pure YAML (or JSON) provide decent workaround to the requested feature, but it does not extend into any SLS renderer combination.

[0xf10e]

Maybe using libucl (BTW: very nice presentation about UCL in FreeBSD) would mitigate some of the problems. The parser can handle things like includes and reuse of assigned variables on its own.
Not as pythonic as YAML but less of a hassle to write than JSON.

[DanyC97]

+1 on proper implementation

[J-Fricke]

Found this trying to set some variables in a pillar file and the import_yaml suggestion by @morsik fits the bill perfectly in my use case.

After reading the comment by @uvsmtid, I simply named my yaml file with .yaml extension to avoid confusion.

[ketzacoatl]

FWIW, reclass handles this use case very gracefully, and I would recommend considering reclass if you need some keys to reference other keys - http://reclass.pantsfullofunix.net/operations.html#parameter-interpolation

[hemebond]

Do we have an example of what kind of output this feature should generate? Functional specs? Each issue asking for this feature is slightly different.

I did some testing to replicate a simple system I had in a previous (homemade) config system: https://gist.github.com/hemebond/3b31f3ed5ba89dfc0af7655a498b8d2b

[ketzacoatl]

FWIW, reclass does this perfectly, and has exactly what the primary use cases want out of the general idea. If you want pillar to be able to interpolate from other pillar, right now, the best way to do so is to use reclass as an ext pillar module. It works great, and actually vastly simplifies your use of pillar (eg for defaults, salt's setup for defining default pillars is really messy in comparison). EDIT: sorry, I didn't mean to sound like a broken record, I overlooked my previous comment immediately above the last, lol..

[hemebond]

I thought reclass only interpolated variables that appeared higher in the hierarchy. Is that not the case? If reclass does everything people need for Pillar interpolation do we need to keep any of these tickets open?

[AndreiPashkin]

@jberkus

My implementation idea is simple: load the pillars in the order given in pillar/top.sls. If a user wants a pillar to refer to another pillar, they need to get the ordering right themselves.

How then, in your opinion, Salt should handle pillar data, provided from a CLI (a.k.a pillar overrides, which are controlled by pillar_roots_override_ext_pillar setting) and also from external pillars? Should CLI pillar be visible for regular pillars? Should subsequent regular pillars override CLI pillar in sequence? Or should CLI pillar be applied at the end and override all pillars - external and regular?

I'm busy preparing a PR with implementation of your solution and I stumbled upon this problems.

[jberkus]

@AndrewPashkin

Beats me, I never got that complex. My Salt setups were relatively simple; I just joined this issue because I wanted a lot of re-use of variables. My ordering suggestion was a way to do "baby steps", and otherwise resolve an order-of-resolution issue which was blocking solutions.

[ketzacoatl]

FWIW, reclass has the code to show us how to construct the tree and walk it.

[AndreiPashkin]

@jberkus, yes, I understand, but your idea implies, that order of when CLI pillars and external pillars are loaded, must also be defined.

Otherwise it would be undefined behaviour, for example user would make some expectations, based on order of his pillars, but somebody could also provide CLI pillars and it would lead to unexpected results and so on.

[jberkus]

Makes sense. Is the ordering currently undefined?

BTW, just so you know: the project which caused me to comment on this issue has been over for more than a year, so I won't be able to test any solution.

[AndreiPashkin]

@jberkus, currently external pillars are always loaded before regular ones and are reference-able from regular ones (I've mistaken in my past comment, pillar_roots_override_ext_pillar only affects final composition, not order of loading). External pillars are also merged with CLI pillars. And CLI pillars also override all pillars in the end.

[jberkus]

Well, sounds like the order of load is defined, then. It's perfectly OK if that ordering is defined by Salt. It's just with the standard pillars where ordering is undefined.

[AndreiPashkin]

@jberkus, hm, yes, you are right, actually.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[ketzacoatl]

salt could use reclass and easily provide this functionality.

[stale]

Thank you for updating this issue. It is no longer marked as stale.

[0xf10e]

Yeah, @ketzacoatl, reclass (GH: madduck/reclass) is probably the best choice here.
Not sure if adding a dependency on reclass is a good idea though.

[ketzacoatl]

Yea, that's a question beyond me, though I don't think it needs to be made an external dependency. Salt could still make use of the code - it's clean, reliable, and robust. As a user, the experience is awesome.

[bbinet]

@ketzacoatl @0xf10e FYI saltclass (which is very similar to reclass) is now directly available in salt from version 2018.3.0.
See: https://docs.saltstack.com/en/latest/topics/releases/2018.3.0.html#new-saltclass-pillar-master-tops-modules

[ketzacoatl]

oh wow... so this is already done then.. we should close this ticket as complete.. no?

[0xf10e]

Sweet, thanks for pointing this out for us, @bbinet!
Don't have the time to follow the development as close as I'd like lately…

[perfecto25]

not fastest performance wise, but can do this by importing pillar as yaml

pillar1.sls

{% set mypillar = "../pillar/" + grains.get('host') + ".sls" %}
{% import_yaml mypillar as yaml %}
{{ yaml.data }}   # abc

/pillar/host1.sls

data: abc

[morsik]

@perfecto25 this is something I proposed 8 years ago... and if you read this issue, you can find that this feature was added 5 years... that's quite gold shovel prize :P