Use JSON import for environment variables

[clementd-fretlink]

clever env import now supports reading a JSON value, a list of {"name": "MY_VAR", "value": "yolo" } objects.

Using JSON instead of an ad-hoc format makes things more robust, especially wrt quotes and newlines. I have tested it on lane-explorer: https://gitlab.in.fretlink.com/fretlink/lane-explorer/-/jobs/156189

The tricky part is that the values need to be provided as strings (as are environment variables, after all). This was done a bit implicitly previously, now this is done explicitly (this also explains why i was not able to use the dict2items filter directly.

Warning

Before this PR, when we provide an env var containing line breaks, they are escaped and transformed into literal \n, that need to be unescaped later. As far as I know, it's only done for the SSH key in ci-commons. With this PR, this step is not useful anymore and can be removed.

[adfretlink]

LGTM

[paulrbr-fl]

I'd rather have the variable manipulation done outside the template. Its usually better to understand the variable manipulation next to the ansible tasks.

[paulrbr-fl]

Warning

Before this PR, when we provide an env var containing line breaks, they are escaped and transformed into literal \n, that need to be unescaped later. As far as I know, it's only done for the SSH key in ci-commons. With this PR, this step is not useful anymore and can b

I am sorry but I don't understand this warning. Where do we “escape and transform line breaks into literal \n”?

[clementd-fretlink]

The escaping was done by the to_json filter here: https://github.com/fretlink/ansible-clever/blob/master/templates/env.j2#L2 in addition to adding quotes, it also escapes special chars

The unescaping is performed by echo -e here: https://github.com/fretlink/continuous-integration-commons/blob/master/dhall/deploy/ansible/clever/DefaultEnv.dhall#L68

[paulrbr-fl]

The escaping was done by the to_json filter here: https://github.com/fretlink/ansible-clever/blob/master/templates/env.j2#L2 in addition to adding quotes, it also escapes special chars

Hm, but that's by nature of JSON values. And we were already transforming any values to be a JSON value. So I still don't understand the difference before and after the PR on the values we provide to clever

[clementd-fretlink]

The difference is that before this PR, we provide a file KEY=value, where value is a JSON literal.
clever-tools treats it as a regular string, so it does not remove the escaping. That's why in the case of the ssh key, we have to use echo -e to remove the escaping.

With this, PR, clever-tools reads the env as JSON values and performs the unescaping itself

[clementd-fretlink]

Using to_json with the KEY=value format was a trick to properly escape quotes and newlines, but it's still a trick. Here, we use json to serialize the env, both on the ansible side and on the clever-tools side

[paulrbr-fl]

The difference is that before this PR, we provide a file KEY=value, where value is a JSON literal.
clever-tools treats it as a regular string, so it does not remove the escaping. That's why in the case of the ssh key, we have to use echo -e to remove the escaping.

With this, PR, clever-tools reads the env as JSON values and performs the unescaping itself

🤯 indeed very well noted 👌 thanks for the explanation.

[clementd-fretlink]

Alright! After yesterday i was a bit scared to modify things further, but now it works well, so 🚀