Finalise standardisations remaining from the `_mapdata` implementation #205

myii · 2020-09-04T20:52:11Z

#203 has been merged but there are outstanding issues that need to be finalised, particularly when standardising across all of the formulas in our organisation (using the ssf-formula). Quoting the relevant comment here:

[1] Use text-based instead of YAML-based comparison? including test(map): verify map.jinja dump using _mapdata state #203 (comment)

No change required (at least for the time being): Just use text-based comparison and update the PRs in other formulas to do the same.

[2] Set up mapdata in _mapdata/init.sls

No change required (at least for the time being): Merge as-is and resolve this when centralising the solution in the ssf-formula.

[3] Remove replacement from _mapdata_spec.rb? including test(map): verify map.jinja dump using _mapdata state #203 (comment)

No change required (at least for the time being): I don't believe there will be many formulas that will need this so no need to add the extra layer of complexity; however, can be resolved when centralising the solution in the ssf-formula.

[4] Use control scope for variables in InSpec?

No change required (at least for the time being): Merge as-is and open up for further discussion; @n-rodriguez, do you know anything about this?

[5] Change commands => command

@baby-gnu, it would be better to get this fixed before merging; perhaps I can add a tiny commit to do this.

[6] Use InSpec shared resources across all formulas?

No change required (at least for the time being): Merge as-is and resolve this when centralising the solution in the ssf-formula.

[7] Replace old platform_finger concatenation with shared library method instead

No change required (at least for the time being): Either resolve in an extra commit or leave for a subsequent PR.

The text was updated successfully, but these errors were encountered:

baby-gnu · 2020-09-05T08:58:42Z

Hello.

For [1], we could add an extra state in _mapdata/init.sls to strip CR from the generated file on windows only?

Looking at file.managed documentation and issues, I found that there is no way to force line ending but I found that opened issue saltstack/salt#14135.

myii · 2020-09-05T16:38:29Z

Hello.

For [1], we could add an extra state in _mapdata/init.sls to strip CR from the generated file on windows only?

Looking at file.managed documentation and issues, I found that there is no way to force line ending but I found that opened issue saltstack/salt#14135.

@baby-gnu The whole discussion started when I was testing this _mapdata implementation with the salt-formula, since it has so many platforms and is tested on Windows as well. Using gsub was the workaround that did it:

Of course, would just need to apply that for Windows only.

baby-gnu · 2020-09-07T07:20:18Z

@baby-gnu The whole discussion started when I was testing this _mapdata implementation with the salt-formula, since it has so many platforms and is tested on Windows as well. Using gsub was the workaround that did it:

yes, we have some internal formulas at work only for windows machines and found an issue this way.

I was wondering what's better between manage the difference in the state or the check. any preference?

myii · 2020-09-07T08:18:58Z

I was wondering what's better between manage the difference in the state or the check. any preference?

@baby-gnu Do you have a working state available? It would be useful to compare both methods.

baby-gnu · 2020-09-07T09:58:02Z

@baby-gnu Do you have a working state available? It would be useful to compare both methods.

You can try the following:

diff --git a/TEMPLATE/_mapdata/init.sls b/TEMPLATE/_mapdata/init.sls
index cfa4837..d253a3c 100644
--- a/TEMPLATE/_mapdata/init.sls
+++ b/TEMPLATE/_mapdata/init.sls
@@ -16,3 +16,11 @@
     - template: jinja
     - context:
         map: {{ mapdata | yaml }}
+
+{%- if grains['os_family'] == 'Windows' %}
+{{ tplroot }}-mapdata-strip-newline:
+  file.replace:
+    - name: {{ output_file }}
+    - pattern: '\r'
+    - repl: ''
+{%- endif %}

Here is the result when applying the TEMPLATE._mapdata a second time:

PC-386675.example.net:
----------
          ID: TEMPLATE-mapdata-dump
    Function: file.managed
        Name: /tmp/salt_mapdata_dump.yaml
      Result: True
     Comment: File /tmp/salt_mapdata_dump.yaml updated
     Started: 11:53:06.906250
    Duration: 109.375 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,13 +1,13 @@
                  -# yamllint disable rule:indentation rule:line-length
                  -# Windows-7
                  ----
                  -added_in_defaults: defaults_value
                  -config: /etc/TEMPLATE
                  -pkg:
                  -  name: TEMPLATE
                  -rootgroup: root
                  -service:
                  -  name: TEMPLATE
                  -subcomponent:
                  -  config: /etc/TEMPLATE-subcomponent-formula.conf
                  -winner: defaults
                  +# yamllint disable rule:indentation rule:line-length
                  +# Windows-7
                  +---
                  +added_in_defaults: defaults_value
                  +config: /etc/TEMPLATE
                  +pkg:
                  +  name: TEMPLATE
                  +rootgroup: root
                  +service:
                  +  name: TEMPLATE
                  +subcomponent:
                  +  config: /etc/TEMPLATE-subcomponent-formula.conf
                  +winner: defaults
----------
          ID: TEMPLATE-mapdata-strip-newline
    Function: file.replace
        Name: /tmp/salt_mapdata_dump.yaml
      Result: True
     Comment: Changes were made
     Started: 11:53:07.015625
    Duration: 0.0 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1,13 +1,13 @@
                  -# yamllint disable rule:indentation rule:line-length
                  -# Windows-7
                  ----
                  -added_in_defaults: defaults_value
                  -config: /etc/TEMPLATE
                  -pkg:
                  -  name: TEMPLATE
                  -rootgroup: root
                  -service:
                  -  name: TEMPLATE
                  -subcomponent:
                  -  config: /etc/TEMPLATE-subcomponent-formula.conf
                  -winner: defaults
                  +# yamllint disable rule:indentation rule:line-length
                  +# Windows-7
                  +---
                  +added_in_defaults: defaults_value
                  +config: /etc/TEMPLATE
                  +pkg:
                  +  name: TEMPLATE
                  +rootgroup: root
                  +service:
                  +  name: TEMPLATE
                  +subcomponent:
                  +  config: /etc/TEMPLATE-subcomponent-formula.conf
                  +winner: defaults

The salt-run state.event pretty=True show the following:

salt/job/20200907095318212067/ret/PC-386675.example.net	{
    "_stamp": "2020-09-07T09:53:21.087071",
    "cmd": "_return",
    "fun": "state.apply",
    "fun_args": [
        "TEMPLATE._mapdata"
    ],
    "id": "PC-386675.example.net",
    "jid": "20200907095318212067",
    "out": "highstate",
    "retcode": 0,
    "return": {
        "file_|-TEMPLATE-mapdata-dump_|-/tmp/salt_mapdata_dump.yaml_|-managed": {
            "__id__": "TEMPLATE-mapdata-dump",
            "__run_num__": 0,
            "__sls__": "TEMPLATE._mapdata",
            "changes": {
                "diff": "--- \n+++ \n@@ -1,13 +1,13 @@\n-# yamllint disable rule:indentation rule:line-length\n-# Windows-7\n----\n-added_in_defaults: defaults_value\n-config: /etc/TEMPLATE\n-pkg:\n-  name: TEMPLATE\n-rootgroup: root\n-service:\n-  name: TEMPLATE\n-subcomponent:\n-  config: /etc/TEMPLATE-subcomponent-formula.conf\n-winner: defaults\n+# yamllint disable rule:indentation rule:line-length\r\n+# Windows-7\r\n+---\r\n+added_in_defaults: defaults_value\r\n+config: /etc/TEMPLATE\r\n+pkg:\r\n+  name: TEMPLATE\r\n+rootgroup: root\r\n+service:\r\n+  name: TEMPLATE\r\n+subcomponent:\r\n+  config: /etc/TEMPLATE-subcomponent-formula.conf\r\n+winner: defaults\r\n"
            },
            "comment": "File /tmp/salt_mapdata_dump.yaml updated",
            "duration": 109.375,
            "name": "/tmp/salt_mapdata_dump.yaml",
            "result": true,
            "start_time": "11:53:06.906250"
        },
        "file_|-TEMPLATE-mapdata-strip-newline_|-/tmp/salt_mapdata_dump.yaml_|-replace": {
            "__id__": "TEMPLATE-mapdata-strip-newline",
            "__run_num__": 1,
            "__sls__": "TEMPLATE._mapdata",
            "changes": {
                "diff": "--- \n+++ \n@@ -1,13 +1,13 @@\n-# yamllint disable rule:indentation rule:line-length\r\n-# Windows-7\r\n----\r\n-added_in_defaults: defaults_value\r\n-config: /etc/TEMPLATE\r\n-pkg:\r\n-  name: TEMPLATE\r\n-rootgroup: root\r\n-service:\r\n-  name: TEMPLATE\r\n-subcomponent:\r\n-  config: /etc/TEMPLATE-subcomponent-formula.conf\r\n-winner: defaults\r\n+# yamllint disable rule:indentation rule:line-length\n+# Windows-7\n+---\n+added_in_defaults: defaults_value\n+config: /etc/TEMPLATE\n+pkg:\n+  name: TEMPLATE\n+rootgroup: root\n+service:\n+  name: TEMPLATE\n+subcomponent:\n+  config: /etc/TEMPLATE-subcomponent-formula.conf\n+winner: defaults\n"
            },
            "comment": "Changes were made",
            "duration": 0.0,
            "name": "/tmp/salt_mapdata_dump.yaml",
            "result": true,
            "start_time": "11:53:07.015625"
        }
    },
    "success": true
}

baby-gnu · 2020-09-07T10:03:18Z

NB: for windows I had to create the /tmp directory manually ;-)

baby-gnu · 2020-09-09T08:31:30Z

I finished with the following code to handle windows minions:

diff --git a/TEMPLATE/_mapdata/init.sls b/TEMPLATE/_mapdata/init.sls
index cfa4837..18e96fc 100644
--- a/TEMPLATE/_mapdata/init.sls
+++ b/TEMPLATE/_mapdata/init.sls
@@ -16,3 +16,23 @@
     - template: jinja
     - context:
         map: {{ mapdata | yaml }}
+
+{%- if grains['os_family'] == 'Windows' %}
+{#- file.directory requires an absolute path with drive letter #}
+{% set system_drive = salt['environ.get']('SYSTEMDRIVE', 'C:') %}
+{% set tmp_dir = system_drive ~ '/tmp' %}
+
+{{ tplroot }}-mapdata-tmp-file-directory:
+  file.directory:
+    - name: {{ tmp_dir }}
+    - require_in:
+      - file: {{ tplroot }}-mapdata-dump
+
+{{ tplroot }}-mapdata-strip-newline:
+  file.replace:
+    - name: {{ output_file }}
+    - pattern: '\r'
+    - repl: ''
+    - require:
+      - file: {{ tplroot }}-mapdata-dump
+{%- endif %}

Unfortunately, I don't know how to run InSpec with my windows machine, so I don't know how to check if it's working correctly.

dafyddj · 2020-09-09T15:55:10Z

You can run InSpec against the Windows Vagrant image used by Kitchen in some other formulas, but it hasn't been implemented in the template-formula yet.

baby-gnu · 2021-01-07T14:12:33Z

I closed [3] and [4] since the latest _mapdata_spec.rb implement them.

myii · 2021-01-07T14:27:37Z

Actually, the rest are also resolved now (so I've ticked them):

[1] Use text-based instead of YAML-based comparison? including test(map): verify map.jinja dump using _mapdata state #203 (comment)

template-formula/test/integration/default/controls/_mapdata_spec.rb

Lines 8 to 16 in d0d9cce

    
           ### Method 
        
           # The steps below for each file appear convoluted but they are both required 
        
           # and similar in nature: 
        
           # 1. The earliest method was to simply compare the files textually but this often 
        
           #    led to false positives due to inconsistencies (e.g. spacing, ordering) 
        
           # 2. The next method was to load the files back into YAML structures and then 
        
           #    compare but InSpec provided block diffs this way, unusable by end users 
        
           # 3. The final step was to dump the YAML structures back into a string to use 
        
           #    for the comparison; this both worked and provided human-friendly diffs

[2] Set up mapdata in _mapdata/init.sls
- https://github.com/myii/ssf-formula/blob/5a5c507434d1687d4437845346ee7a76a240f6fd/ssf/files/default/formula/_mapdata/init.sls#L8-L10
[6] Use InSpec shared resources across all formulas?
- https://github.com/myii/ssf-formula/blob/51160d18db22988a3dfbb952d1556a5e0fdb8acb/ssf/formulas.yaml#L120

Leaving this issue open for now since there may be another issue encountered when finalising situations with the v5 map.jinja conversion in these initial formulas. Let's close this issue once it has been successfully established in this formula.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finalise standardisations remaining from the `_mapdata` implementation #205

Finalise standardisations remaining from the `_mapdata` implementation #205

myii commented Sep 4, 2020 •

edited

Loading

baby-gnu commented Sep 5, 2020 •

edited

Loading

myii commented Sep 5, 2020

baby-gnu commented Sep 7, 2020

myii commented Sep 7, 2020

baby-gnu commented Sep 7, 2020

baby-gnu commented Sep 7, 2020

baby-gnu commented Sep 9, 2020

dafyddj commented Sep 9, 2020 •

edited

Loading

baby-gnu commented Jan 7, 2021

myii commented Jan 7, 2021

Finalise standardisations remaining from the _mapdata implementation #205

Finalise standardisations remaining from the _mapdata implementation #205

Comments

myii commented Sep 4, 2020 • edited Loading

baby-gnu commented Sep 5, 2020 • edited Loading

myii commented Sep 5, 2020

baby-gnu commented Sep 7, 2020

myii commented Sep 7, 2020

baby-gnu commented Sep 7, 2020

baby-gnu commented Sep 7, 2020

baby-gnu commented Sep 9, 2020

dafyddj commented Sep 9, 2020 • edited Loading

baby-gnu commented Jan 7, 2021

myii commented Jan 7, 2021

Finalise standardisations remaining from the `_mapdata` implementation #205

Finalise standardisations remaining from the `_mapdata` implementation #205

myii commented Sep 4, 2020 •

edited

Loading

baby-gnu commented Sep 5, 2020 •

edited

Loading

dafyddj commented Sep 9, 2020 •

edited

Loading