readiness

[boneskull]

I want to use this. When can I use this? 😄

[boneskull]

Ref: mochajs/mocha#4427

[dominykas]

If you have an org to keep the forks in, and are happy to maintain those forks manually - probably today? I'll try to come up with a roadmap in the coming weeks.

[dominykas]

OK, I see your use case. It's probably somewhat doable today, yes, although some manual setup required. I'm happy to try to set things up and see how well it stacks up? Would be useful to have some concrete use cases to evaluate as we're building this - including yours.

[dominykas]

Responded on the mocha issue - shall we continue the discussion there?

Otherwise "ready" is a very relative concept... I did create https://github.com/pkgjs/wiby/projects/1 to track the overarching theme of making this actually useful and suitable for humans.

Unless there's objections - will close this issue in a week or so?

[boneskull]

I've responded over in that issue. not sure what you're talking about re "forks"

[dominykas]

not sure what you're talking about re "forks"

At the moment wiby requires push permissions on the repositories you want to test against, so if it's a third party dep you want to test - you need to fork it. I wrote that before I read through your issue - it does not apply to you, I guess.

[boneskull]

hmm, so, to be clear, I'm trying to check that e.g. the interaction between mocha and the latest version of chai does not break when I make a change to mocha. would that require a fork of chai? if so, why?

[dominykas]

would that require a fork of chai? if so, why?

So you could achieve this in two ways - one would be to clone chai, modify package.json, npm install, npm test. There's several tools that do a similar thing already (but I'm not sure there's an obvious popular leading one that does that).

The idea behind wiby is that it tries to execute all the tests the same way as if you opened a PR against the dependent repo - and that means using that repo's CI setup: running in all the node versions that it supports, executing the full build matrix, possibly executing some additional integration-y jobs. The simplest way to do it is to actually open up a PR against that repo (or at least push a branch) - and that's what wiby tries to do.

So, if you don't have push access to chai - you need to fork it, but even with push access you might not want to open PRs that just test mocha (although in a lot of cases just pushing a branch is enough, so it's less disruptive in terms of notifications).

interaction between mocha and the latest version of chai

Depending on the type of that interaction you want to test, wiby might not be an ideal tool for this. mocha is a dev dependency of chai. wiby should help answer "if I release this commit of mocha - will chai still work?" (Will I Break You) - I realize that a lot of people use mocha and chai together, but checking if they work together (rather than as a dependecy) is a slightly different use case (which I haven't thought through yet).

[dominykas]

(I'm aware express doesn't use chai, but let's pretend it does)

To test chai + mocha still works, instead of forking/testing in chai itself, you'd probably want to fork express, and link wip of mocha into there, and see if the tests of express still pass.

[boneskull]

I agree that the "best" way to test is to open a PR against the repo, yes. But, consider that CITGM doesn't even do that. I would also be concerned about causing an explosion of used build minutes. Further, the need to have a bunch of forks laying around and keeping them updated (or creating them on-the-fly) seems like even more overhead. I am not sure I'd want to do this for Mocha, and would need to be convinced that it's really worth the tradeoffs.

Regarding chai and mocha, maybe that was a poor example. I would be interested in whether or not a change in Mocha broke Chai's tests. Ensuring the usage of Chai within a Mocha context is a different problem, and one that I'd want to hand-write tests for (we already have this).

If Wiby's aim is to run tests in CI, then--and it's not up for debate--I wonder if a "mode" in which it essentially does a CITGM-like "clone and test" would be on the table?

@BethGriggs I know your team had some involvement in this at one point, and I was wondering if there was anything I should be aware of in terms of goals/use-cases/users?

[dominykas]

I wonder if a "mode" in which it essentially does a CITGM-like "clone and test" would be on the table?

Honestly, I'm not sure we'd need to build that, because there's already alternatives that do that?

I was just notified of https://github.com/xing/canarist today.

That said - I'm not strictly against it either.

As for build minutes - if you own the forks, then you essentially use up the same amount of minutes, if you're going to try to test in all the relevant Node.js versions / full matrix? And you're probably interested if Mocha broke Chai's tests in Node 10 as well as in 12, 14, 15?

[boneskull]

Honestly, I'm not sure we'd need to build that, because there's already alternatives that do that?

There are, but I don't know of any that are in active development or wide use. The PR to extend CITGM in this way has stalled; fugl hasn't been published in a year. canarist seems tailored to workspaces/monorepos...

As for build minutes - if you own the forks, then you essentially use up the same amount of minutes, if you're going to try to test in all the relevant Node.js versions / full matrix? And you're probably interested if Mocha broke Chai's tests in Node 10 as well as in 12, 14, 15?

Well, wouldn't it run whatever the CI is configured to run? So if Express used Mocha and we wanted to make sure we didn't break Express, it'd run Express' CI suite? Or would we be expected to modify the CI configuration to run only relevant tests? e.g., I would skip an ESLint check since that has nothing to do with Mocha.

[dominykas]

So if Express used Mocha and we wanted to make sure we didn't break Express, it'd run Express' CI suite?

Yeah, so that's what wiby intends to do. Maybe I misunderstood what you said, but if you want to run Express' CI suite, then you need to use their CI config (Travis or Actions or whatever), and that is tied to their repo anyways, so a fork (if you don't have/can't use direct push access) is a simple way of doing that? And it's the job of wiby to hide away the complexity of managing the fork, or at least that's the goal.

e.g., I would skip an ESLint check since that has nothing to do with Mocha.

Some of the wild ideas around this was even go as far as fetching, say, top 1000 of your dependents and then picking random 10 and testing against those. So no - modifying configs manually is probably a no go.

[boneskull]

Yes, I understand. It does sound like it'd potentially use a lot of build minutes.

What I'd like to understand is this: is running the complete CI suite of project X is worth the added overhead? If CITGM does not do this, and has historically (IMO) done a good job of not unknowingly breaking userland, is that not "good enough?"

It sounded like what was being asked for in the issue you linked above was "CITGM, except generalized", which is different than this strategy. We know, then, that there is a need for a CITGM-like tool. Do maintainers want what wiby is intending to provide?

[dominykas]

I think it depends on the use case. Deprecating an old version of node and starting to use new features will most definitely break dependents. Then there's larger ecosystems to consider, namely stringing multiple PRs from different repos together.

And then there's the fact that yes, I want what wiby is intending to provide, so I'm building it.

Not saying that wiby should or shouldn't support the simple "test single version directly" case, just that I'm focusing on this use case for now.