Add support for yaml config files

[matfax]

What would you like Renovate to be able to do?

JSON isn't considered the best choice for configuration files anymore. This blog post on lucidchart explains some of the reasons for it. Most services (e.g., GitHub actions) use YAML as the new standard. In my opinion, YAML support for configuration files would make a helpful addition to Renovate. The configuration would be more comprehensible and also easier to edit, especially when using GitHub's online edit feature.

Did you already have any implementation ideas?

Since yaml is a superset of json, the config file could be parsed using the yaml library, and then, equally processed as the json-derived config object.

Is this a feature you'd be interested in implementing yourself?

Depends on your feedback and the level of interest in this feature.

[rarkins]

No, let's stick with JSON. We have a lot of config examples in our documentation and it would be painful to maintain both. In addition we have the JSON schema now which can be used for validation in IDEs. YAML is far from perfect too, and while the blog post may argue that JSON's strictness is a weakness, I think that YAML's lack of strictness can be equally problematic.

[rarkins]

Reopening this for reconsideration.

We could support YAML config even if all our examples are JSON. People choosing to use YAML would need to accept that.

Ideally one day all JSON code snippets in our markdown docs could be processed to generate the equivalent in YAML and then display both in docs (switchable).

We also need to consider the impact of YAML on presets. We currently check for default.json and renovate.json, which already means some 404s. We wouldn't want more try/fail/try attempts. Maybe if they're YAML then the full filename with extension needs to be referenced.

[davidspek]

As I just discussed with @rarkins, I'd be willing to manually go through the docs to update them with YAML examples. People that would like to migrate from JSON to YAML can use yq to convert their file easily with yq eval -P sample.json > sample.yaml.

[rarkins]

What I was hoping for visually in our docs is something like this mockup (ignore the languages/example):

Ideally JSON would be our single source of truth - therefore not many docs need changing - but we can transform them on the way to HTML so that tabbed examples can be shown, similar to the above.

[davidspek]

Where did you get that example from? I've been trying to find just that so I can try and find how to create it online.

[rarkins]

I found it in a meta Stack Overflow post discussing adding the feature to SO. But better news - it seems like such tabs are supported by Docusaurus, which we were planning to switch to.

Exactly what we need: https://v2.docusaurus.io/docs/markdown-features/#multi-language-support-code-blocks

[davidspek]

That indeed looks like the perfect fit, what a coincidence.

[rarkins]

So, approximate steps:

1. Migrate to docusaurus. Note: I'm starting to think it should remain in its own repo, because it will need to pull from multiple repos, not just this one.
2. Use .mdx extension instead of .md. Not sure if a bulk rename simply works or if there's any downsides?
3. Add post-processing for our markdown files to convert all ` json blocks into Docusaurus tabulated code blocks with a YAML equivalent (automatically generated)

This way users can keep writing simple markdown with JSON blocks as is today, but the docs published to docs.renovatebot.com have multi-language examples.

One more thing: need to think about the admin config (config.js, config.json) too.

[nejch]

@rarkins I would love to see support for yaml!

You don't even need to wait to migrate to docusaurus, as mkdocs supports this natively, so this shouldn't be a blocker to continue with the feature.

See renovatebot/renovatebot.github.io#65 for a quick implementation of a post-processor with mkdocs (it's a few lines of code really), and below a short demo:

renovate-yaml-2021-06-27_13.51.04.mp4

The caveat is that examples need to be valid json (so currently a few are skipped), I think this would be the case with any post-processor anyway.

[rarkins]

Wow, cool! I'd like all our examples to be fully valid JSON anyway, so would be happy to add further linting to try to ensure it.

I'm currently on vacation so struggling to keep up with the usual volume of issues and PRs. @viceice what do you think?

[viceice]

Sounds good. but before we add the docs we need to allow the yaml config files.

Ideas:

1. We should force file ending for preset config files, so we don't need to more api requests to find files.
2. Maybe the platform apis have some filters, so we can get all files starting with the filename /path prefix and then use the returned name and fetch contents. that way we need 2 api calls instead of one before and we don't need file extensions.

[nejch]

Re: filtering repo files, @viceice I'll just give you what I know for GitLab.

Sadly on gitlab the most useful endpoint (search repo blobs) is a premium feature: https://docs.gitlab.com/ee/api/search.html#scope-blobs

Second best, before actually looping over all possible filepaths, might be the repo tree endpoint:
https://docs.gitlab.com/ee/api/repositories.html#list-repository-tree

But again, if there are a lot of files in the repo that are returned before a renovate config file is found, the pagination might trigger more requests than just looping over ~10 files. This is better if you leave out recursive, but then you need a 2nd call to list files in the .gitlab/ directory as well (to support .gitlab/renovate.json).

So there would need to be a balance, maybe benchmark based on the average repo tree size. Just some thoughts on options :)

[rarkins]

I think if we cache the found config file name then it should be ok to have a small impact on the first run because it will be 0 impact indefinitely after that

[viceice]

What about preset filenames?

[rarkins]

Good point. For backwards compatibility, no extension means .json. For .yaml or .json5 the extension must be included

[nejch]

I think if we cache the found config file name then it should be ok to have a small impact on the first run because it will be 0 impact indefinitely after that

Is this something that's basically already done at least for per-repo config?

renovate/lib/workers/repository/init/merge.ts

Lines 27 to 38 in 9184207

	const cache = getCache();
	let { configFileName } = cache;
	if (configFileName) {
	let configFileParsed = await platform.getJsonFile(configFileName);
	if (configFileParsed) {
	if (configFileName === 'package.json') {
	configFileParsed = configFileParsed.renovate;
	}
	return { configFileName, configFileParsed };
	}
	logger.debug('Existing config file no longer exists');
	}

[viceice]

Good point. For backwards compatibility, no extension means .json. For .yaml or .json5 the extension must be included

Can we cache the resolved preset file names? that way we can suppress extension for preset too. 🤔

[rarkins]

Depends whether you consider the current lack of file extensions a feature or a bug :)

Less a user needs to type the better. But then we should try to avoid ambiguity and confusion too.

BTW such caching could already be added today for the default.json/renovate.json check. Question is for how long? Eg 24 hours? Never, as long as the cached file name resolves?

[viceice]

My idea waas to cache those in the repo json cache file, where we already have the repo config filename.

I think we can cache the filenames as long as they resolve.

We should also prefer json over yaml, if a user has both available. Should be documented of cause.

[rarkins]

So if there are 100 repos referencing a particular preset then we'd "try/fail" that preset once per downstream repo but then cache it in each downstream repo after that?

Slightly inefficient but on the upside it resolves any concerns about caching private results

[JuliusHenke]

Are there any updates on plans to support renovate.yaml for the repo specific config? The self-hosted admin config already works great with YAML and I love using it. Adding renovate.yaml support would make it more consistent and easier to work with.
This was also proposed in: #10682

[Kurt-von-Laven]

JSON schemas work just as well with YAML as they do with JSON since the two languages are essentially different representations of the same underlying structure. VSCode users get autocomplete, documentation, etc. from schemas that have been added to JSONSchema for example.

In addition we have the JSON schema now which can be used for validation in IDEs.

[rarkins]

This topic came up again recently, so I will add a recent opinion: I don't want to support this unless a regular contributor to Renovate wants to add it and support it in the longer term. While one-off PRs are usually appreciated, in this case it would be like gifting a puppy - we'd be left behind to clean up after it and look after it all its life.