Creating a downstream impact testing group for ansible-lint

[ssbarnea]

One thing that v5 release teached me is that regardless how many pre-releases you make and how much you advertise changes made in upcoming versions, most users will bother test them before the release is made.

So I am planning to gather a limited (10?) group of power users of the linter, lets call it friends of the ansible-lint which can be used a early testers of the changes me to the linter.

Each early-tester should provide one or two actively maintained open-source repositories on which linter is supposed to run and pass without errors.

I will create an integration pipeline on ansible-lint itself named impact which will clone each of these repositories and run the proposed change on them. If it fails, we will fail the job.

Obviously that some of the proposed changes on the linter will uncover problems that need to be fixed on these repositories by either by disabling specific rules or by fixing the problems.

I would not mind to make a pull-request to fix the introduced problem when needed but I do expect support from the owners on addressing these and more important they should provide feedback on these changes.

Please let me know if you want to join the early testers program, the current list of repos I plan to test with will be listed below. Do limit to one, max two repositories as the scope is to have a small but representative test ground.

Proposed repositories and contacts:

• https://github.com/geerlingguy/ansible-role-mysql.git @geerlingguy - standalone role
• https://github.com/devroles/ansible_collection_system @greg-hellings
• https://opendev.org/openstack/tripleo-ansible.git @ssbarnea
• https://github.com/konstruktoid/ansible-docker-rootless @konstruktoid
• https://github.com/konstruktoid/ansible-role-hardening @konstruktoid
• https://opendev.org/zuul/zuul-jobs @ssbarnea - delayed until proposed fix gets merged in (required multiple changes, including porting a custom rule)

[geerlingguy]

I'd be happy to be in that group—and if there's anything you need from me in terms of help getting it set up let me know. I would also be willing to add an additional workflow that could run on ansible-lint devel or something (with ignored failures).

[konstruktoid]

I can sign up to be in that group.
And as @geerlingguy said, if you want I'll add an additional workflow as well.

Two standalone roles:

• https://github.com/konstruktoid/ansible-role-hardening
• https://github.com/konstruktoid/ansible-docker-rootless (this will be converted to FQCN next week)

[juju4]

I can also add few repositories.

But I think the problem is more the visibility of new errors change.
Many(Most?) people don't track pre-releases, only whatever pip is serving.

• Either documentation encourages to have use pre-release at least for some repositories or doing both pre-release and release testing with maybe the former in warn mode only. Maybe in https://ansible-lint.readthedocs.io/en/latest/usage.html
• Either error lint changes are first implemented as warn in one or few releases

Also molecule lint step does not make warning/notice really visible, so less chance it will be reviewed.
I don't think that molecule, github action and others have an easy way for this.

maybe in pipeline execution

pip install ansible-lint --pre
ansible-lint || true
pip install ansible-lint
ansible-lint

but that means doing install stuff during test and reviewing carefully output.

[konstruktoid]

Also molecule lint step does not make warning/notice really visible, so less chance it will be reviewed.

But showing warnings and notices is up to the linting tool, e.g:

INFO     Running default > lint
COMMAND: set -e
ansible-lint --exclude .git --exclude .github --exclude tests/
yamllint -d "{extends: default, ignore: .git*, rules: {line-length: {level: warning}}}" .
[...]
./tasks/main.yml
  19:81     warning  line too long (112 > 80 characters)  (line-length)
  23:81     warning  line too long (86 > 80 characters)  (line-length)
  24:81     warning  line too long (108 > 80 characters)  (line-length)
  38:81     warning  line too long (89 > 80 characters)  (line-length)
[...]

[juju4]

If you look and expand the corresponding molecule step for it yes
but I would like an option in CI to report/email/hook on warnings like errors.
Because many/most won't look at pipeline log details unless it fails IMHO

[greg-hellings]

I would also volunteer my personal repo:
https://github.com/devroles/ansible_collection_system

and my primary work repo
https://github.com/oasis-roles/ansible_collection_system

[ssbarnea]

While working on implementing the test pipeline, I encountered an issue with nesting repos that affected your repo, mainly because you did not use a config file at root. I ended up proposing #1436 -- take a look and tell me if you see if it could bring regressios.

[greg-hellings]

Hmmm. Are you able to pull submodules when you do the checkout? If so, we can add a symlink in the base of my repo from ".ansible-lint" to "tests/ansible-lint" so that this test remains up to date on what my repo is targeting.

[greg-hellings]

Actually I just double-checked and the git Ansible module automatically checks out submodules when it does a fresh clone, so I'm going to go with adding that symlink so you can use it during checking

[ssbarnea]

Please take a look at #1434 change which is supposed to add an eco testing pipeline that runs the new linter with some repositories and let me know what you think.

Apparently the approach works because it seems to enable me to find new bugs.

[tadeboro]

The https://github.com/sensu/sensu-go-ansible can also be used.

[ssbarnea]

Based of more than an year of using eco pipeline, I am going to make some small changes to the way we run linter, basically in order to limit the regressions I observed so far, where repos broke that pipeline.

From now on, we should also add a GHA pipeline that is using ansible-lint from main branch on these repos, so the testing with be reciprocated (work both ways).

We can either use main branch directly or ansible-lint-action, which is not really main but it is updated often enough to be considered similar. The trick here is to avoid pinning ansible-lint version. That pipeline is supposed to be optional, so it should not prevent these repos from merging changes even if it reports errors, same as we do on ansible-lint.

[ssbarnea]

As part of #2136 we now send github notifications to repository owners about changes that modify the outcome of running ansible-lint on them, so they would know to either advocate against making that change or update their code to fix the problem.

As I am still the contact for two of the repositories: zuul-jobs and tripleo-ansible, I need to find replacement as I am no longer involved with any of the projects and fully overbooked. Who wants to help on this? I only need to add your github username there. We can add as many people we want to https://github.com/ansible/ansible-lint/blob/main/test/eco/CODENOTIFY file, so don't be shy! ;)

[TristanCacqueray]

You can sign me up. Thanks!