ci: properly configure smoke test

[G-Rath]

whoops

[AriPerkkio]

The action doesn't actually pick these .ts configurations by default since action parameter has default value for .js config: https://github.com/AriPerkkio/eslint-remote-tester-run-action/blob/e72eaaa36788a76ef9044271bebbbd83377e7d6d/action.yml#L14-L17

So this needs to be specified here:

    eslint-remote-tester-config: smoke-test/eslint-remote-tester.config.ts

[G-Rath]

Ah yeah I wondered if that might be the case 👍

[AriPerkkio]

Suggested change

	"eslint-remote-tester": "^2.1.0",
	"eslint-remote-tester": "^2.1.1",

[G-Rath]

@AriPerkkio thanks for the review - tbh I'm sort of on the fence about sticking with this now that I know you've actually been running the same workflow for our plugin and from what I can tell we've never actually had an error; it seems like a bit of a waste of power to be running the same ~6 hour workflow twice...

[G-Rath]

(I'm going to merge now just so it's fixed, but still thinking about maybe reverting)

[G-Rath]

@AriPerkkio random other question: does the remote tester support anyway to handle linting repositories multiple times with different rule configurations? My thinking is about how rule logic can be changed with options, so running against e.g. jest/all wouldn't cover all of the possible code the rules could run.

I can't find anything in the docs that look like a way to do this - if I've not missed anything, let me know if you think this could be useful and I can open a new issue to continue discussing.

[github-actions]

🎉 This PR is included in version 25.3.4 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[AriPerkkio]

@G-Rath this path is still incorrect 😄

-          eslint-remote-tester-config: smoke-test/eslint-remote-tester.config.ts
+          eslint-remote-tester-config: eslint-remote-tester.config.ts

#1002 (comment)

Edit: Right there's an open PR for this, great!

[G-Rath]

Yup

[AriPerkkio]

does the remote tester support anyway to handle linting repositories multiple times with different rule configurations?

At the moment, no. I've also had this idea before but didn't proceed with implementing it. I think the implementation wouldn't be that difficult though.
This is the spot where we are reading the config.eslintrc which can be either a single configuration, or function returning one. https://github.com/AriPerkkio/eslint-remote-tester/blob/30803c6b628da08515c6fbbe49065153dd727982/lib/engine/worker-task.ts#L212-L218
It could simply return a list of configuration which would then be looped in the rest of workerTask below.

Currently it is only possible to initialize config.eslintrc based on repository, e.g. checking whether project has tsconfig.json and enabling certain rules: https://github.com/AriPerkkio/eslint-remote-tester/blob/30803c6b628da08515c6fbbe49065153dd727982/ci/plugin-configs/typescript-eslint-eslint-plugin.config.js#L10.

[AriPerkkio]

tbh I'm sort of on the fence about sticking with this now that I know you've actually been running the same workflow for our plugin and from what I can tell we've never actually had an error

I've been running the jest/all for long time and this has been one of the most stable plugin I've been testing. Though there have been some other plugins as well which have been running successfully for months and then suddenly caused random crashes. eslint-plugin-regexp is a good example here.

Running these tests in repository where main branch contains unreleased changes makes a lot sense, e.g. eslint-plugin-react. However this project uses automatic releasing so that's not perfect reasoning here.

I think you'll get the most from these tests if eslintrc.rules are configured extensively to cover as much code blocks as possible. This is something I've imagined to happen in projects themselves.
Also bringing the tests closer to the actual plugin projects is always better since I don't have to be building minimal repros and reporting the bugs anymore. 😄
You'll get issues reported by Github bot automatically in the project.