Allow master config defining *static custom* data values.

[uvsmtid]

This was initially named as "custom static master grains" in Salt-users group due to similarity with "custom static minion grains".

The main difference between the two:

• master one is centralized and manually defined in single /etc/salt/master.
• minion ones are distributed and manually defined in multiple /etc/salt/minion.

Let's assume for now this custom data is accessed by name config (similar to grains or pillar) in all Salt SLS files.

Why

The irreplaceable property of config is the ability to parameterize SLS files processed by master:

• Any top files (for example, /srv/salt/top.sls, /srv/pillar/top.sls, ...).
• SLS files for runners executed by salt-run, (for example, see orchestration).

All these files processed by master (not minion). And master does not have static custom configuration values like grains for minions (master has grains but they cannot contain static custom values). Therefore, compared to minions, templates rendered by master are limited in sources for parameters (neither custom grains exists, nor other data like pillars because it is yet to be compiled).

Use case

Let's assume several identical server farms managed by their own master. They all use the same sources for their states (file_roots) and data (pillar_roots). However, each server farms needs slightly different configuration data in pillar provided from the sources:

# /srv/pillar/top.sls
base:
    '*':
        - {{ config['project'] }}.project_specific_data

Using grains instead of config would not work because this is pillar's top file which is rendered by master while master grains cannot have custom values like project.

The solution would be defining single custom value inside each master configuration file:

# /etc/salt/master
...
config:
    project: enigma
...

[terminalmage]

This has been possible for quite some time, via config.get. So long as the key you are looking up is not defined in grains, pillar, or a minion config file, the master config file will return a value for the specified key. It also can traverse multi-level dictionaries.

[terminalmage]

Here's an example:

[root@test ~]# salt \* config.get config
cent6:
    ----------
    foo:
        bar
[root@test ~]#
[root@test ~]# salt \* config.get config:foo
cent6:
    bar
[root@test ~]# cat /etc/salt/master
interface: 127.0.0.1

config:
  foo: bar
[root@test ~]#

[uvsmtid]

@terminalmage thanks!
I'm surprised the answer was there all this time.

Actually, I was even close - I noticed before that master does not complain if any arbitrary data is defined inside its config file. But I was only able to access this data on minions that time (using pillar_opts option in master config and pillar on minion side). Function config.get meets the requirement.

Use case rewritten

This is a rewritten use case from above:

# /etc/salt/master
...
whatever_key:
    project: enigma
...

# /srv/pillar/top.sls
base:
    '*':
        - {{ salt['config.get']('whatever_key:project') }}.project_specific_data

I consider the feature request is closed at least for me.

Note

The only thing which could be done is emphasis or warning in the documentation regarding grains that they are minion-only.

It took me some time to realize why {{ grains['custom_key'] }} failed with salt-run in orchestration and top files (corrected: if rendered by minion, not master, custom static values in grains are available in top files). The reason is obvious now: master-side software couldn't access grains which are minion-only. Until one understands the mechanics, the failure appears very counter-intuitive when master-side command fails while using the same grains in minion-side is perfectly OK.

[terminalmage]

Custom grains should be available in the top file. How are you defining them?

[uvsmtid]

My bad - I wasn't precise. The problem with example was posted in the beginning of this topic.
To be correct: they are not available in top files for master-side commands (i.e. salt-run) only.
So, I've adjusted the invalid note above.

[basepi]

So the remaining piece that needs to be done for this issue is adding master-side availability of the grains for orchestration files?

[uvsmtid]

@basepi Function config.get solves everything for me. And I don't feel strong need for another solution.

However, If someone needs the same functionality he may also find it difficult to discover this function. So, I would expect to see this for consistency in the far future. The most intuitive implementation is to make master-side grains be actually available through the same grains variable:

• If it is master-side software, grains provide values from /etc/salt/master.
• If it is minion-side software, grains provide values from /etc/salt/minion.

And, thanks to config.get or pillar_opts option, minions can also access master grains, for example:

{{ salt['config.get']('grains:custom_key') }}
...
{{ pillar['master']['grains']['custom_key'] }}

Since there is a workaround, if the issue is closed, I won't complain.

[basepi]

I think I'm going to close it for now. If another user hits this use case, we may decide to implement it.

[uvsmtid]

After almost a year I got somewhat addicted to use

salt['config.get']('any_parent_key:any_child_key')

even beyond the top files (as I initially needed). I try to limit its use but this is the only way to parameterise pillar (jinja) template files by centralized data (which is Salt master config file in this case) - see example

I think it is remotely related to #6955 (impossible to read data from one pillar by another) because I effectively use Salt master config to inject and access variables into pillar template file. If I was able to get these variables from another pillar (not from Salt master config), I wouldn't need to resort to salt['config.get'].

[Z9n2JktHlZDmlhSvqc9X2MmL3BwQG7tk]

I think publishing of master config to all the minions is a security issue and stupid feature. Why do minions must see master config ? Why not to publish master's private key and /etc/shadow ?
There must be a feature to add some data to pillar without whole config publishing and without assigning to any minion - just some master data. Also it must not be done via master config file, so we will not need to restart salt-master.
What this can be used for ? For example for setting roles to minions on master side. Due to security reasons minions should not tell master who they are, they should tell only "I am minion XYZ, give me my configuration scenarios".
This is my opinion. What do you think ?

[uvsmtid]

"There must be a feature to add some data to pillar without whole config publishing and without assigning to any minion - just some master data."

@Z9n2JktHlZDmlhSvqc9X2MmL3BwQG7tk, what you probably mean by "without assigning to any minion" is available for all minions.

I've already mentioned issue #6955 in my previous comment above - specifically, that comment there. So, workaround by exposing Salt master config file (discussed above) which I used before is no longer needed (and, by the way, this issue was closed). Now, based on the comment (again) loading a file with data (expressed in YAML, or JSON, or ...) into a pillar allows resolving all raised issues:

• It does not expose Master config - the data is in pillar. Therefore, there is no potential security problems.
• Because such data files are loaded through pillars, they are evaluated by master on every usage - this drops the need to restart Salt master after each modification of the data files.
• Because such data files are loaded through pillars, its data can be made available to all minions unconditionally "without assigning to any minion"by assigning specific pillar key (e.g. loaded from an *.sls file which, in turn, loads that YAML/JSON/... data file) to ALL * minions.

Take a look at the few examples in follow-up comments from the comment there (again) to make it less abstract.

I also find it a bit hacky (it's definitely not a first class feature) but the discovery solves small-scale problems now before advent of the perfect future. Personally, what I think is sure is that there is no reason for yet another data source (in addition to grains and pillars). If you look closely into "just some master data" and its use case defined by you, I don't see how this is different from a pillar which is simply assigned to all * minions. There should only be more generic flexibility to work with pillar files (like ease of accessing data from pillars in other pillars = #6955).

Let me know if I miss any point.