Feature/template override and files switch #19

baby-gnu · 2019-02-06T16:04:12Z

This is the TOFS pattern from @moreda with the modifications I did in systemd-formula.

aboe76 · 2019-02-06T16:50:55Z

@baby-gnu nice documentation.

myii · 2019-02-07T00:12:31Z

TOFS_pattern.md

+
+Among other functions, the _master_ (or _salt-master_) serves files to the _minions_ (or _salt-minions_). The [file_roots](http://docs.saltstack.com/en/latest/ref/file_server/file_roots.html) is the list of directories used in sequence to find a file when a minion requires it: the first match is served to the minion. Those files could be [state files](http://docs.saltstack.com/en/latest/topics/tutorials/starting_states.html) or configuration templates, among others.
+
+Using Saltstack is a simple and effective way to implement configuration management, but even in a [non multitenant](http://en.wikipedia.org/wiki/Multitenancy) scenario, it's not a good idea to generally accesible some data (e.g. the database password in our [Zabbix](http://www.zabbix.com/) server configuration file or the private key of our [Nginx](http://nginx.org/en/) TLS certificate).


accesible => accessible.

myii · 2019-02-07T00:13:09Z

TOFS_pattern.md

+{% set default_servers = ['0.ubuntu.pool.ntp.org',
+                          '1.ubuntu.pool.ntp.org',
+                          '2.ubuntu.pool.ntp.org',
+                          '3.ubuntu.pool.ntp.org']}


Missing %:

'3.ubuntu.pool.ntp.org'] %}

myii · 2019-02-07T00:13:34Z

TOFS_pattern.md

+{% endfor %}
+```
+
+This way we are localy **overriding the template files** offered by the formula in order to make a more complex adaptation. Of course, this could be applied as well to any of the files, including the state files.


localy => locally.

myii · 2019-02-07T00:14:25Z

template/macros.jinja

+
+      /etc/yyy/zzz.conf:
+        file:
+          - managed


Use file.managed instead.

myii · 2019-02-07T00:14:45Z

TOFS_pattern.md

+
+      /etc/yyy/zzz.conf:
+        file:
+          - managed


Use file.managed instead.

myii · 2019-02-07T00:14:59Z

TOFS_pattern.md

+
+      /etc/yyy/zzz.conf:
+        file:
+          - managed


Use file.managed instead.

myii · 2019-02-07T00:15:16Z

template/macros.jinja

+
+      /etc/yyy/zzz.conf:
+        file:
+          - managed


Use file.managed instead.

noelmcloughlin

LGTM - hope this formula evolves. thanks @baby-gnu

vutny

Great review @myii

myii · 2019-02-07T08:55:17Z

@vutny Thanks for the compliment.

I don't intend to hijack this discussion but this PR has reminded me of a more general situation in respect to these SaltStack formulas. With a number of active contributors involved in this PR discussion, hopefully someone can suggest a better place to talk about the points I mention below:

In discussions on the SaltStack Community Slack workspace, I've found some resistance to these SaltStack formulas.
Main issues appear to be that these formulas are too heavily reliant on using pillars (leading to excess load on the master) alongside general quality issues.

I'm not expecting anywhere near a complete solution to this. However, I believe it would be useful to compare ideas, to identify any (small) improvements that realistically could be made and maintained.

baby-gnu · 2019-02-07T12:11:18Z

Thanks to all the contribution, I'll integrate them and push them as soon as possible (and I found an issue in macros.jinja ;-)).

Regards.

* TOFS_pattern.md: add myself as modifier. Trim trailing whitespaces. Separate titles from first paragraph.

The example could not be used as-is. This commit improve conformity to formula conventions. * TOFS_pattern.md: add missing commas “,” in “map.jinja” and extra one to ease the addition of new entries. Import “map.jinja” in “init.sls” and “conf.sls”. Declare descriptive state IDs. Use the “module.function” notation. Use the “name” parameter.

The prefix was used for 2 purposes: - define the pillar prefix where to lookup “:files_switch”. It supports the colon “:” separator to lookup in pillar subtree like “foo:bar” - define the path prefix where to look for “files/”, It did not support separator to lookup inside directory tree. This patch only replace any colon “:” with “/” when looking up “files/” directory, with the “foo:bar” prefix: - lookup “foo:bar:files_switch” pillar to get list of grains to match - lookup files under “salt://foo/bar/files/” * TOFS_pattern.md: document the new use of “prefix” supporting colon “:”. * template/macros.jinja: transform any colon “:” in “prefix” by slash “/” to lookup files.

When the files are “full path” with leading slash “/”, the generated URL contain a double slash because of the join. * template/macros.jinja: remove leading slash before joining parts. * TOFS_pattern.md: mirror changes of “macros.jinja”.

The “template” is kept during rendering. * TOFS_pattern.md: add “template” to rendered state. * template/macros.jinja: ditoo.

aboe76 · 2019-02-08T19:43:12Z

@myii I agree hijacking a discussion for the greater good...:)
I would love to see these salt-formula's become something like a community maintenance project.
something allong the line of vox pupuli https://voxpupuli.org/ but then for salt

myii · 2019-02-08T20:05:57Z

@aboe76 I just had a quick look at https://voxpupuli.org/ and I'm impressed by the effort they have put in over the years. That far surpasses what I was envisaging! I'm intrigued by the idea -- but I don't think it's right to continue off-topic for this PR. Is there a more appropriate place for this discussion to be continued?

aboe76 · 2019-02-08T20:12:32Z

@myii : https://groups.google.com/forum/#!forum/salt-formulas ??

myii · 2019-02-08T20:32:25Z

OK, @aboe76 has contacted me in Slack and we're using the main formulas channel there to continue this discussion. Please feel free to join us there.

baby-gnu · 2019-02-08T20:34:38Z

@myii Sorry but I do not use Slack for “political” reasons (we can't even see discussions without an account). I already accept to use GitHub to contribute to projects I like ;-)

Thanks for making this discussion alive.

Regards.

myii · 2019-02-08T20:36:05Z

@baby-gnu I'm not particularly interested in keeping it on Slack. I'm just not sure Google Groups is the easiest place either. Can you suggest another place?

myii · 2019-02-08T20:39:59Z

@baby-gnu @aboe76 How about something like a Matrix (riot.im) or Telegram group?

baby-gnu · 2019-02-08T21:57:55Z

@myii @aboe76 I'm not sure I understood the purpose of this discussion, but if the SaltStack community wants to use Slack to organise the formula community I will not complain, I'll just be off the discussions.

I'm used to mailing-list and IRC, I make few exceptions to my avoidance of centralised services ;-)

aboe76 · 2019-02-08T23:16:38Z

@baby-gnu salt has an irc channel on freenode, maybe create a salt-formulas channel there as well
and maybe link slack channel formulas with that one, so you and others won't be left out...

aboe76 · 2019-02-08T23:19:17Z

@baby-gnu merged, thanks for you contributions, and please continu your work.

myii · 2019-02-08T23:19:32Z

@baby-gnu Matrix is decentralised as well. Besides, bridging between Matrix and IRC is simple, albeit a little ugly (in presentation). I'm good either way but there have been some very useful points made so far.

myii · 2019-02-08T23:20:34Z

@baby-gnu Thanks for the work. @aboe76 Thanks for the merge.

myii · 2019-02-09T02:05:41Z

All of the initial discussions have been collected on a wiki page. There have been some very interesting points made so I encourage you to have a look over it. I believe it contains a good basis for making improvements across the whole of SaltStack Formulas.

baby-gnu · 2019-02-09T07:37:15Z

@aboe76 thanks for the merge. By the way, there is already a #salt-formulas IRC channel for github.com/salt-formulas, looks to be a separated from saltstack-formula ;-)

See you.

javierbertoli · 2019-02-10T21:56:13Z

github.com/salt-formulas is an interesting project I saw in the past and to which I get back from time to time for inspiration. The first time I saw it was when I moved from chef to salt and was trying to find the community's formulas. I found that group and this.

The thing that prevented me for using salt-formulas and moved me to participate in saltstack-formulas is that their quickstart bootstrap is anything but simple. They write formulas in a consistent way, but at the cost (IMHO) of a too complex initial setup and a heavy inter-dependence between formulas (as I understand them, maybe I'm wrong). I think some ideas can be learned from their ecosystem too.

aboe76 · 2019-02-10T22:08:14Z

@baby-gnu the github.com/salt-formulas are nice,
they rely on reclass or the newer saltclass to do the heavy lifting of pillar data merging or overwriting.

And I agree with @javierbertoli we can learn a few things from how they develop the formulas and stay consistent.

myii · 2019-02-11T06:19:29Z

Ongoing discussions have been captured on another wiki page. More importantly, a SaltStack Formulas project has been created and populated with all of the discussion points so far.

myii · 2019-02-13T17:50:33Z

CC: @baby-gnu @vutny @aboe76 @noelmcloughlin @javierbertoli @alxwr.

We're live!!! Thanks to @gtmanfred, #saltstack-formulas on Freenode is now bridged to the Slack #formulas room. Note, this is not #salt-formula, which is linked to https://github.com/salt-formulas as @baby-gnu mentioned above.

Any suggestions for decent logging of the IRC channel? There's some cracking discussions and I don't want them lost into the ether (or is that aether?). Nor do I fancy manually keeping logs in the wiki! I'm sure I've come across something in the past but I'll have to search again.

In any case, no excuses. See you on the other side.

BTW, @baby-gnu: there have been a number of conversations about this TOFS pattern approach. I can't do it justice. Your feedback would be invaluable.

myii · 2019-02-13T18:12:44Z

CC: @baby-gnu @vutny @aboe76 @noelmcloughlin @javierbertoli @alxwr.

Sorry for the mass pingouts again but now there really is no excuse. IRC logging is in place, so you never have to miss any discussions: https://freenode.logbot.info/saltstack-formulas.

baby-gnu added 2 commits February 6, 2019 16:09

Import TOFS pattern from Zabbix formula

2dd659f

Switch template.config to TOFS pattern

784ce72

aboe76 approved these changes Feb 6, 2019

View reviewed changes

aboe76 requested review from javierbertoli, noelmcloughlin and myii February 6, 2019 16:51

myii reviewed Feb 7, 2019

View reviewed changes

template/macros.jinja Outdated

/etc/yyy/zzz.conf:

file:

- managed

Copy link

Member

myii Feb 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use file.managed instead.

myii reviewed Feb 7, 2019

View reviewed changes

TOFS_pattern.md Outdated

/etc/yyy/zzz.conf:

file:

- managed

Copy link

Member

myii Feb 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use file.managed instead.

myii reviewed Feb 7, 2019

View reviewed changes

TOFS_pattern.md Outdated

/etc/yyy/zzz.conf:

file:

- managed

Copy link

Member

myii Feb 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use file.managed instead.

myii reviewed Feb 7, 2019

View reviewed changes

template/macros.jinja Outdated

/etc/yyy/zzz.conf:

file:

- managed

Copy link

Member

myii Feb 7, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use file.managed instead.

noelmcloughlin approved these changes Feb 7, 2019

View reviewed changes

vutny reviewed Feb 7, 2019

View reviewed changes

baby-gnu added 5 commits February 7, 2019 15:52

Cosmetics modification of TOFS pattern documentation

8bbab14

* TOFS_pattern.md: add myself as modifier. Trim trailing whitespaces. Separate titles from first paragraph.

Add an example for “ntp” of the use of “files_switch”

24eaeb7

baby-gnu force-pushed the feature/Template-Override-and-Files-Switch branch from 224915c to 8bd2157 Compare February 7, 2019 16:18

Examples must be consistent

312fcdb

The “template” is kept during rendering. * TOFS_pattern.md: add “template” to rendered state. * template/macros.jinja: ditoo.

aboe76 merged commit 95e5e37 into saltstack-formulas:master Feb 8, 2019

baby-gnu deleted the feature/Template-Override-and-Files-Switch branch February 9, 2019 17:42

myii mentioned this pull request Feb 11, 2019

Establish template-formula as the go-to resource for all formula development #21

Open

71 tasks

alxwr mentioned this pull request Mar 19, 2019

remove ng stuff saltstack-formulas/apache-formula#255

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/template override and files switch #19

Feature/template override and files switch #19

baby-gnu commented Feb 6, 2019

aboe76 commented Feb 6, 2019

myii Feb 7, 2019

myii Feb 7, 2019

myii Feb 7, 2019

myii Feb 7, 2019

myii Feb 7, 2019

myii Feb 7, 2019

myii Feb 7, 2019

noelmcloughlin left a comment

vutny left a comment

myii commented Feb 7, 2019

baby-gnu commented Feb 7, 2019

aboe76 commented Feb 8, 2019

myii commented Feb 8, 2019

aboe76 commented Feb 8, 2019

myii commented Feb 8, 2019

baby-gnu commented Feb 8, 2019

myii commented Feb 8, 2019

myii commented Feb 8, 2019

baby-gnu commented Feb 8, 2019

aboe76 commented Feb 8, 2019

aboe76 commented Feb 8, 2019

myii commented Feb 8, 2019 •

edited

Loading

myii commented Feb 8, 2019

myii commented Feb 9, 2019

baby-gnu commented Feb 9, 2019

javierbertoli commented Feb 10, 2019

aboe76 commented Feb 10, 2019

myii commented Feb 11, 2019 •

edited

Loading

myii commented Feb 13, 2019 •

edited

Loading

myii commented Feb 13, 2019


		Among other functions, the _master_ (or _salt-master_) serves files to the _minions_ (or _salt-minions_). The [file_roots](http://docs.saltstack.com/en/latest/ref/file_server/file_roots.html) is the list of directories used in sequence to find a file when a minion requires it: the first match is served to the minion. Those files could be [state files](http://docs.saltstack.com/en/latest/topics/tutorials/starting_states.html) or configuration templates, among others.

		Using Saltstack is a simple and effective way to implement configuration management, but even in a [non multitenant](http://en.wikipedia.org/wiki/Multitenancy) scenario, it's not a good idea to generally accesible some data (e.g. the database password in our [Zabbix](http://www.zabbix.com/) server configuration file or the private key of our [Nginx](http://nginx.org/en/) TLS certificate).

Feature/template override and files switch #19

Feature/template override and files switch #19

Conversation

baby-gnu commented Feb 6, 2019

aboe76 commented Feb 6, 2019

myii Feb 7, 2019

Choose a reason for hiding this comment

myii Feb 7, 2019

Choose a reason for hiding this comment

myii Feb 7, 2019

Choose a reason for hiding this comment

myii Feb 7, 2019

Choose a reason for hiding this comment

myii Feb 7, 2019

Choose a reason for hiding this comment

myii Feb 7, 2019

Choose a reason for hiding this comment

myii Feb 7, 2019

Choose a reason for hiding this comment

noelmcloughlin left a comment

Choose a reason for hiding this comment

vutny left a comment

Choose a reason for hiding this comment

myii commented Feb 7, 2019

baby-gnu commented Feb 7, 2019

aboe76 commented Feb 8, 2019

myii commented Feb 8, 2019

aboe76 commented Feb 8, 2019

myii commented Feb 8, 2019

baby-gnu commented Feb 8, 2019

myii commented Feb 8, 2019

myii commented Feb 8, 2019

baby-gnu commented Feb 8, 2019

aboe76 commented Feb 8, 2019

aboe76 commented Feb 8, 2019

myii commented Feb 8, 2019 • edited Loading

myii commented Feb 8, 2019

myii commented Feb 9, 2019

baby-gnu commented Feb 9, 2019

javierbertoli commented Feb 10, 2019

aboe76 commented Feb 10, 2019

myii commented Feb 11, 2019 • edited Loading

myii commented Feb 13, 2019 • edited Loading

myii commented Feb 13, 2019

myii commented Feb 8, 2019 •

edited

Loading

myii commented Feb 11, 2019 •

edited

Loading

myii commented Feb 13, 2019 •

edited

Loading